[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:00 dist-test-slave-8bgs krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:00 dist-test-slave-8bgs krb5kdc[399](info): set up 2 sockets
Sep 27 21:07:00 dist-test-slave-8bgs krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 21:07:01.963533 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:02.073117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:02.150280 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:02.150484 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:02.150506 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:02.151641 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:02.151681 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:02.151695 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:02.151708 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:02.151759 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:02.153151 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:02.153388 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:02.153635 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:02.158635 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.158692 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:02.158632 412 server_base.cc:1034] running on GCE node
W20260927 21:07:02.158672 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:02.159034 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:02.159327 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:02.160482 412 hybrid_clock.cc:648] HybridClock initialized: now 1790543222160477 us; error 32 us; skew 500 ppm
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:02.162685 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:02.163157 412 webserver.cc:460] Webserver started at http://127.0.98.62:39757/ using document root <none> and password file <none>
I20260927 21:07:02.163398 412 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:02.163462 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:02.163607 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:02.164605 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "087196ce5d624c8ebc7f2f8110e71395"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "b3c7b3070558267b4a9ea7eb358bd354"
server_key_iv: "70ff9621206652ef153ed58bbf0b0b9b"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.164947 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "087196ce5d624c8ebc7f2f8110e71395"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "b3c7b3070558267b4a9ea7eb358bd354"
server_key_iv: "70ff9621206652ef153ed58bbf0b0b9b"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.166224 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:02.166922 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.167114 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:02.167210 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "087196ce5d624c8ebc7f2f8110e71395"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "b3c7b3070558267b4a9ea7eb358bd354"
server_key_iv: "70ff9621206652ef153ed58bbf0b0b9b"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.167287 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:02.186717 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:02.188486 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:02.188643 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:02.192502 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39199
I20260927 21:07:02.192552 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39199 every 8 connection(s)
I20260927 21:07:02.192914 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:02.193758 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:02.195974 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395: Bootstrap starting.
I20260927 21:07:02.196497 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:02.196741 480 log.cc:824] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:02.197320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 412
I20260927 21:07:02.197407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:02.197943 392 external_mini_cluster.cc:1428] Setting key 99ed992d2f720c5160b48dc11fa1f97e
I20260927 21:07:02.198997 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395: No bootstrap required, opened a new log
I20260927 21:07:02.200551 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087196ce5d624c8ebc7f2f8110e71395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260927 21:07:02.200685 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:02.200707 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087196ce5d624c8ebc7f2f8110e71395, State: Initialized, Role: FOLLOWER
I20260927 21:07:02.200851 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087196ce5d624c8ebc7f2f8110e71395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260927 21:07:02.200937 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:02.200968 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:02.201012 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:02.201838 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087196ce5d624c8ebc7f2f8110e71395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260927 21:07:02.201925 480 leader_election.cc:302] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087196ce5d624c8ebc7f2f8110e71395; no voters:
I20260927 21:07:02.202119 480 leader_election.cc:288] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:02.202150 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:02.202255 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [term 1 LEADER]: Becoming Leader. State: Replica: 087196ce5d624c8ebc7f2f8110e71395, State: Running, Role: LEADER
I20260927 21:07:02.202337 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087196ce5d624c8ebc7f2f8110e71395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } }
I20260927 21:07:02.202374 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:02.202636 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087196ce5d624c8ebc7f2f8110e71395. Latest consensus state: current_term: 1 leader_uuid: "087196ce5d624c8ebc7f2f8110e71395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087196ce5d624c8ebc7f2f8110e71395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } } }
I20260927 21:07:02.202716 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:02.202647 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087196ce5d624c8ebc7f2f8110e71395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087196ce5d624c8ebc7f2f8110e71395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39199 } } }
I20260927 21:07:02.202907 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:02.202965 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:02.203536 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:02.205057 490 catalog_manager.cc:1347] Generated new cluster ID: 08f199c1d068460baccb87387131bccf
I20260927 21:07:02.205111 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:02.215070 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.198726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.98.62:39199)
0927 21:07:02.198953 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:02.198963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:02.199246 (+ 283us) server_negotiation.cc:407] Connection header received
0927 21:07:02.200888 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:02.200895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:02.200951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:02.201090 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:02.202840 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.204205 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.205600 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.206162 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.206261 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.210454 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:02.210458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:02.210458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:02.210468 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:02.213148 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.213594 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.213597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.213597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.213714 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.213977 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.213978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.213978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.214282 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:02.214446 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:02.214708 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:07:02.214853 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":119,"threads_started":1}
I20260927 21:07:02.215857 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:02.216321 490 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:02.222355 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087196ce5d624c8ebc7f2f8110e71395: Generated new TSK 0
I20260927 21:07:02.222651 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:02.247388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:02.347456 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:02.347651 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:02.347677 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:02.348819 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:02.348866 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:02.348892 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:02.350297 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:02.350528 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:02.350785 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:02.355741 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.355723 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.355727 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:02.356000 506 server_base.cc:1034] running on GCE node
I20260927 21:07:02.356245 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:02.356492 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:02.357694 506 hybrid_clock.cc:648] HybridClock initialized: now 1790543222357686 us; error 33 us; skew 500 ppm
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:02.360248 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:02.360641 506 webserver.cc:460] Webserver started at http://127.0.98.1:37151/ using document root <none> and password file <none>
I20260927 21:07:02.360872 506 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:02.360936 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:02.361083 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:02.362143 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "bed87a961dc046ddb46ec19acdcad587"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "986054b2b79702e28e54a0666e7eb04b"
server_key_iv: "f635e56851e91fc75948afe76d62c9fe"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.362502 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bed87a961dc046ddb46ec19acdcad587"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "986054b2b79702e28e54a0666e7eb04b"
server_key_iv: "f635e56851e91fc75948afe76d62c9fe"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.363837 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:02.364534 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.364686 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.364781 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "bed87a961dc046ddb46ec19acdcad587"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "986054b2b79702e28e54a0666e7eb04b"
server_key_iv: "f635e56851e91fc75948afe76d62c9fe"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.364856 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:02.378625 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:02.380319 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:02.380470 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:02.380673 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:02.380950 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:02.380998 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.381032 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:02.381059 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.386998 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35495
I20260927 21:07:02.387045 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35495 every 8 connection(s)
I20260927 21:07:02.387348 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:02.392141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 506
I20260927 21:07:02.392241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:02.392383 392 external_mini_cluster.cc:1428] Setting key b24a7e989dbd28c8a47e8a4c44549a61
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:02.396485 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.388006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33933 (local address 127.0.98.62:39199)
0927 21:07:02.388080 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:02.388083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:02.388249 (+ 166us) server_negotiation.cc:407] Connection header received
0927 21:07:02.388740 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:02.388741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:02.388753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:02.388804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:02.389684 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.390186 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.391177 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.391376 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.391458 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.393556 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:02.393558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:02.393558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:02.393564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:02.395204 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.395576 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.395577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.395578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.395638 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.395874 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.395875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.395875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.395986 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:02.396058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:02.396317 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:07:02.396396 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:07:02.396672 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.387970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39199 (local address 127.0.98.1:33933)
0927 21:07:02.388137 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:02.388169 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:02.388633 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:02.388882 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:02.388886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:02.389407 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:02.389584 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:02.389589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.390276 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.390277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:02.391050 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:02.391052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.391467 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.391468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:02.391581 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.391981 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:02.391988 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:02.393460 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:02.395270 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:02.395276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:02.395279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:02.395490 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:02.395706 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:02.395707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:02.395708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:02.395811 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:02.396048 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:02.396054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:02.396187 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:07:02.396296 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:07:02.396446 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260927 21:07:02.397148 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39199
I20260927 21:07:02.397233 635 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:02.397406 635 heartbeater.cc:511] Master 127.0.98.62:39199 requested a full tablet report, sending...
I20260927 21:07:02.397950 444 ts_manager.cc:194] Registered new tserver with Master: bed87a961dc046ddb46ec19acdcad587 (127.0.98.1:35495)
I20260927 21:07:02.398880 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33933
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:02.415033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:02.494081 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:02.494277 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:02.494300 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:02.495446 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:02.495488 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:02.495515 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:02.496873 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:02.497046 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:02.497208 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:02.501873 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.501907 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.501907 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:02.501916 640 server_base.cc:1034] running on GCE node
I20260927 21:07:02.502288 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:02.502601 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:02.503750 640 hybrid_clock.cc:648] HybridClock initialized: now 1790543222503738 us; error 36 us; skew 500 ppm
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:02.506094 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:02.506552 640 webserver.cc:460] Webserver started at http://127.0.98.2:46749/ using document root <none> and password file <none>
I20260927 21:07:02.506768 640 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:02.506834 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:02.506966 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:02.508020 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "b78736f112a6474b8d9b1785d7e00711"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "ae67604a905bad4eb1493a64248c84ad"
server_key_iv: "551e4d908329328e8519df1672a597fd"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.508390 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b78736f112a6474b8d9b1785d7e00711"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "ae67604a905bad4eb1493a64248c84ad"
server_key_iv: "551e4d908329328e8519df1672a597fd"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.509646 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:02.510299 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.510468 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:02.510560 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "b78736f112a6474b8d9b1785d7e00711"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "ae67604a905bad4eb1493a64248c84ad"
server_key_iv: "551e4d908329328e8519df1672a597fd"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.510627 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:02.526010 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:02.528156 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:02.528307 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:02.528517 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:02.528803 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:02.528853 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.528893 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:02.528923 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.534960 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36391
I20260927 21:07:02.535000 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36391 every 8 connection(s)
I20260927 21:07:02.535316 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:02.539657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 640
I20260927 21:07:02.539760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:02.539928 392 external_mini_cluster.cc:1428] Setting key 844d4a60ba7187649b63104e0ea6ae87
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:02.544301 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.535955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50657 (local address 127.0.98.62:39199)
0927 21:07:02.536001 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:02.536003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:02.536159 (+ 156us) server_negotiation.cc:407] Connection header received
0927 21:07:02.536603 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:02.536604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:02.536613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:02.536668 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:02.537484 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.537970 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.538861 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.539036 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.539118 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.541554 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:02.541556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:02.541556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:02.541562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:02.542898 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.543372 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.543373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.543373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.543422 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.543695 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.543695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.543696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.543788 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:02.543836 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:02.544175 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 21:07:02.544240 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:02.544530 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.535919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39199 (local address 127.0.98.2:50657)
0927 21:07:02.536056 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:02.536095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:02.536527 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:02.536722 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:02.536725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:02.537238 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:02.537416 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:02.537420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.538040 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.538041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:02.538792 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:02.538793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.539142 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.539143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:02.539236 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.539695 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:02.539702 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:02.541471 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:02.542967 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:02.542973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:02.542975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:02.543304 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:02.543483 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:02.543484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:02.543484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:02.543631 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:02.543850 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:02.543855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:02.544020 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:07:02.544136 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:07:02.544300 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260927 21:07:02.544994 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39199
I20260927 21:07:02.545080 769 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:02.545251 769 heartbeater.cc:511] Master 127.0.98.62:39199 requested a full tablet report, sending...
I20260927 21:07:02.545583 444 ts_manager.cc:194] Registered new tserver with Master: b78736f112a6474b8d9b1785d7e00711 (127.0.98.2:36391)
I20260927 21:07:02.545938 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50657
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:02.562991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:02.642379 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:02.642565 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:02.642587 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:02.643782 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:02.643823 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:02.643873 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:02.645251 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:02.645471 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:02.645691 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:02.650468 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.650446 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:02.650517 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:02.650861 774 server_base.cc:1034] running on GCE node
I20260927 21:07:02.651047 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:02.651291 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:02.652437 774 hybrid_clock.cc:648] HybridClock initialized: now 1790543222652429 us; error 25 us; skew 500 ppm
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:02.654660 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:02.655001 774 webserver.cc:460] Webserver started at http://127.0.98.3:32853/ using document root <none> and password file <none>
I20260927 21:07:02.655221 774 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:02.655265 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:02.655362 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:02.656278 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca850af76e6640dfaf63a97a1d694db6"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "9abbed1165b244876bd8df625b97763b"
server_key_iv: "f462c97823738676cb09553e60658486"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.656591 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca850af76e6640dfaf63a97a1d694db6"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "9abbed1165b244876bd8df625b97763b"
server_key_iv: "f462c97823738676cb09553e60658486"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.657758 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:07:02.658375 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.658517 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:02.658594 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "ca850af76e6640dfaf63a97a1d694db6"
format_stamp: "Formatted at 2026-09-27 21:07:02 on dist-test-slave-8bgs"
server_key: "9abbed1165b244876bd8df625b97763b"
server_key_iv: "f462c97823738676cb09553e60658486"
server_key_version: "encryptionkey@0"
I20260927 21:07:02.658644 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:02.672556 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:02.674196 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:02.674299 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:02.674467 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:02.674690 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:02.674717 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.674737 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:02.674750 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:02.681212 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33233
I20260927 21:07:02.681257 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33233 every 8 connection(s)
I20260927 21:07:02.681531 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:02.687500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 774
I20260927 21:07:02.687593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:02.687780 392 external_mini_cluster.cc:1428] Setting key b091c73b4f986ead41f2f54871bd5c11
I20260927 21:07:02.690172 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.682197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33127 (local address 127.0.98.62:39199)
0927 21:07:02.682252 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:02.682265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:02.682425 (+ 160us) server_negotiation.cc:407] Connection header received
0927 21:07:02.682905 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:02.682907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:02.682918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:02.682963 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:02.683870 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.684345 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.685298 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.685501 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.685579 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.687508 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:02.687509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:02.687510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:02.687516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:02.689104 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.689467 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.689467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.689468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.689515 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.689701 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.689701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.689702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.689801 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:02.689848 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:02.690058 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 21:07:02.690112 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:07:02.690317 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.682173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39199 (local address 127.0.98.3:33127)
0927 21:07:02.682329 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:02.682358 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:02.682807 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:02.683039 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:02.683043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:02.683612 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:02.683780 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:02.683784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.684438 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.684439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:02.685199 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:02.685200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.685586 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.685587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:02.685697 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.686095 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:02.686102 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:02.687419 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:02.689166 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:02.689171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:02.689172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:02.689377 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:02.689565 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:02.689566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:02.689566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:02.689655 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:02.689855 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:02.689859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:02.689960 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 21:07:02.690036 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:07:02.690143 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260927 21:07:02.690688 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39199
I20260927 21:07:02.690775 903 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:02.690961 903 heartbeater.cc:511] Master 127.0.98.62:39199 requested a full tablet report, sending...
I20260927 21:07:02.691289 444 ts_manager.cc:194] Registered new tserver with Master: ca850af76e6640dfaf63a97a1d694db6 (127.0.98.3:33233)
I20260927 21:07:02.691622 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33127
I20260927 21:07:02.699661 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:02.721066 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.713936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.0.98.1:35495)
0927 21:07:02.714092 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:02.714101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:02.714111 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:02.714133 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:02.714135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:02.714147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:02.714229 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:02.714758 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.715653 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.716325 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.716605 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.716655 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.718650 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:02.718657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:02.718658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:02.718664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:02.720148 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.720423 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.720425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.720426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.720476 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.720606 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.720606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.720607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.720718 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:02.720841 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:02.720913 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:07:02.720989 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260927 21:07:02.721426 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49336
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:02.741348 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:02.735081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.0.98.1:35495)
0927 21:07:02.735155 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:02.735157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:02.735223 (+ 66us) server_negotiation.cc:407] Connection header received
0927 21:07:02.735267 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:02.735268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:02.735278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:02.735333 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:02.735856 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.736320 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.736972 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:02.737166 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:02.737214 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:02.739226 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:02.739227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:02.739227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:02.739233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:02.740542 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.740807 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.740808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.740808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.740853 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:02.740985 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:02.740985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:02.740986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:02.741058 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:02.741135 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:02.741196 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:07:02.741248 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:07:02.741871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 506
I20260927 21:07:02.746707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 640
I20260927 21:07:02.752018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 774
I20260927 21:07:02.756011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 412
2026-09-27T21:07:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1816 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[917](info): set up 2 sockets
Sep 27 21:07:02 dist-test-slave-8bgs krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 21:07:03.770632 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:03 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:03.877931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:03.956980 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:03.957212 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:03.957247 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:03.958379 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:03.958446 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:03.958472 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:03.958498 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:03.958526 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:03.959983 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:03.960237 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:03.960453 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:03.965006 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:03.965011 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:03.965022 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:03.965158 930 server_base.cc:1034] running on GCE node
I20260927 21:07:03.965437 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:03.965692 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:03.966853 930 hybrid_clock.cc:648] HybridClock initialized: now 1790543223966843 us; error 41 us; skew 500 ppm
Sep 27 21:07:03 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:03.969273 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:03.969710 930 webserver.cc:460] Webserver started at http://127.0.98.62:39821/ using document root <none> and password file <none>
I20260927 21:07:03.969959 930 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:03.970022 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:03.970151 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:03.971194 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "5c01c39c36604d04a1de4a1bf60d7de2"
format_stamp: "Formatted at 2026-09-27 21:07:03 on dist-test-slave-8bgs"
server_key: "2061be07d8762f71f33a4914f181a91e"
server_key_iv: "3f158ee07ce90058f543e825f310c2f4"
server_key_version: "encryptionkey@0"
I20260927 21:07:03.971529 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c01c39c36604d04a1de4a1bf60d7de2"
format_stamp: "Formatted at 2026-09-27 21:07:03 on dist-test-slave-8bgs"
server_key: "2061be07d8762f71f33a4914f181a91e"
server_key_iv: "3f158ee07ce90058f543e825f310c2f4"
server_key_version: "encryptionkey@0"
I20260927 21:07:03.972780 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:03.973453 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:03.973604 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:03.973701 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "5c01c39c36604d04a1de4a1bf60d7de2"
format_stamp: "Formatted at 2026-09-27 21:07:03 on dist-test-slave-8bgs"
server_key: "2061be07d8762f71f33a4914f181a91e"
server_key_iv: "3f158ee07ce90058f543e825f310c2f4"
server_key_version: "encryptionkey@0"
I20260927 21:07:03.973765 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:03.993345 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:03.995059 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:03.995297 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:03.999199 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43689
I20260927 21:07:03.999231 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43689 every 8 connection(s)
I20260927 21:07:03.999562 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:04.000339 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:04.002125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 930
I20260927 21:07:04.002211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:04.002372 392 external_mini_cluster.cc:1428] Setting key 0a4b942df25c055bd910633edbab8334
I20260927 21:07:04.002693 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2: Bootstrap starting.
I20260927 21:07:04.003355 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:04.003656 998 log.cc:824] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:04.004392 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2: No bootstrap required, opened a new log
I20260927 21:07:04.006374 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260927 21:07:04.006543 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:04.006569 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c01c39c36604d04a1de4a1bf60d7de2, State: Initialized, Role: FOLLOWER
I20260927 21:07:04.006709 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260927 21:07:04.006779 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:04.006821 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:04.006867 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:04.007562 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260927 21:07:04.007676 998 leader_election.cc:302] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c01c39c36604d04a1de4a1bf60d7de2; no voters:
I20260927 21:07:04.007838 998 leader_election.cc:288] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:04.007921 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:04.008087 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:04.008106 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c01c39c36604d04a1de4a1bf60d7de2, State: Running, Role: LEADER
I20260927 21:07:04.008241 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260927 21:07:04.008563 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c01c39c36604d04a1de4a1bf60d7de2. Latest consensus state: current_term: 1 leader_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } } }
I20260927 21:07:04.008669 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:04.008563 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c01c39c36604d04a1de4a1bf60d7de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } } }
I20260927 21:07:04.008764 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:04.008878 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:04.009361 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:04.010963 1010 catalog_manager.cc:1347] Generated new cluster ID: d380f2b9fffb4c30b2009ed54665cfbc
I20260927 21:07:04.011016 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:04.013249 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.002857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.98.62:43689)
0927 21:07:04.003021 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.003030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.003061 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:07:04.003571 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.003587 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.003634 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.003784 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.004500 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.005364 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.006466 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.007047 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.007197 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.010017 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.010021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.010022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.010032 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.011779 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.012140 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.012143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.012144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.012231 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.012435 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.012437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.012437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.012662 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.012796 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.012913 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.013045 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260927 21:07:04.020347 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:04.021046 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:04.025607 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c01c39c36604d04a1de4a1bf60d7de2: Generated new TSK 0
I20260927 21:07:04.025761 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:04.051748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:04.152648 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:04.152878 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:04.152918 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:04.154053 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:04.154115 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:04.154161 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:04.155627 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:04.155874 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:04.156090 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:04.161878 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:04.161886 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:04.161873 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:04.162170 1024 server_base.cc:1034] running on GCE node
I20260927 21:07:04.162379 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:04.162616 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:04.163767 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790543224163753 us; error 32 us; skew 500 ppm
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:04.166302 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:04.166788 1024 webserver.cc:460] Webserver started at http://127.0.98.1:35581/ using document root <none> and password file <none>
I20260927 21:07:04.167025 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:04.167088 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:04.167251 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:04.168385 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "02fa204385db4c62a9defe0786e09b52"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "e521d94395e8cc865eb8d051d9f11fbc"
server_key_iv: "9be0e11f837f1ecaab2b037387236e64"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.168751 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02fa204385db4c62a9defe0786e09b52"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "e521d94395e8cc865eb8d051d9f11fbc"
server_key_iv: "9be0e11f837f1ecaab2b037387236e64"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.170135 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:04.170847 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.171041 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:04.171154 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "02fa204385db4c62a9defe0786e09b52"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "e521d94395e8cc865eb8d051d9f11fbc"
server_key_iv: "9be0e11f837f1ecaab2b037387236e64"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.171236 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:04.180423 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:04.182574 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:04.182726 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:04.182945 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:04.183281 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:04.183336 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.183374 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:04.183403 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.189181 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35973
I20260927 21:07:04.189239 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35973 every 8 connection(s)
I20260927 21:07:04.189522 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:04.196724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1024
I20260927 21:07:04.196800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:04.197001 392 external_mini_cluster.cc:1428] Setting key cf0bf369bfc2e6ac7492fa7bf3db3596
I20260927 21:07:04.198318 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.190158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35169 (local address 127.0.98.62:43689)
0927 21:07:04.190214 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.190216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.190539 (+ 323us) server_negotiation.cc:407] Connection header received
0927 21:07:04.191038 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.191039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.191051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.191145 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.192036 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.192539 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.193503 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.193713 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.193797 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.195733 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.195734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.195735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.195742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.196972 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.197409 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.197410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.197411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.197465 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.197697 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.197698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.197698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.197812 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.197884 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.198146 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.198227 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:07:04.198510 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.190144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43689 (local address 127.0.98.1:35169)
0927 21:07:04.190327 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.190359 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.190941 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.191217 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.191220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.191760 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:04.191941 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.191946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.192638 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.192639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.193406 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.193408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.193800 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.193801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.193914 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.194339 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:04.194347 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:04.195643 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:04.197058 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:04.197065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:04.197067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:04.197342 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:04.197527 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:04.197528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:04.197528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:04.197634 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:04.197874 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:04.197880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:04.198023 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.198124 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.198273 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260927 21:07:04.199158 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43689
I20260927 21:07:04.199258 1153 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:04.199442 1153 heartbeater.cc:511] Master 127.0.98.62:43689 requested a full tablet report, sending...
I20260927 21:07:04.199918 962 ts_manager.cc:194] Registered new tserver with Master: 02fa204385db4c62a9defe0786e09b52 (127.0.98.1:35973)
I20260927 21:07:04.200500 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35169
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:04.219920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:04.297509 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:04.297740 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:04.297778 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:04.298914 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:04.298974 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:04.299021 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:04.300415 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:04.300666 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:04.300880 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:04.305682 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:04.305644 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:04.305653 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:04.305879 1158 server_base.cc:1034] running on GCE node
I20260927 21:07:04.306097 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:04.306341 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:04.307489 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790543224307479 us; error 34 us; skew 500 ppm
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:04.309958 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:04.310350 1158 webserver.cc:460] Webserver started at http://127.0.98.2:36985/ using document root <none> and password file <none>
I20260927 21:07:04.310559 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:04.310601 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:04.310756 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:04.311803 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "84be5e513bd346409cb9508f91a97f3c"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "9c7f9d5fa5e21bccd7cbefd837053b8e"
server_key_iv: "6e347b9e649b0b5756755dccba3090c6"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.312152 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84be5e513bd346409cb9508f91a97f3c"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "9c7f9d5fa5e21bccd7cbefd837053b8e"
server_key_iv: "6e347b9e649b0b5756755dccba3090c6"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.313395 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:04.314045 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.314220 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:04.314308 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "84be5e513bd346409cb9508f91a97f3c"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "9c7f9d5fa5e21bccd7cbefd837053b8e"
server_key_iv: "6e347b9e649b0b5756755dccba3090c6"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.314359 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:04.329119 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:04.331243 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:04.331351 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:04.331516 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:04.331771 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:04.331808 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.331830 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:04.331841 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.337158 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44213
I20260927 21:07:04.337212 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44213 every 8 connection(s)
I20260927 21:07:04.337505 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:04.344098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1158
I20260927 21:07:04.344190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:04.344391 392 external_mini_cluster.cc:1428] Setting key b655b7758fc831e6fde1c5f21d2f11a4
I20260927 21:07:04.346200 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.338187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43743 (local address 127.0.98.62:43689)
0927 21:07:04.338245 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.338247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.338427 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:07:04.338904 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.338905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.338915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.338957 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.339862 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.340340 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.341244 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.341427 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.341473 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.343346 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.343347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.343347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.343353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.344862 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.345309 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.345310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.345310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.345364 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.345631 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.345631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.345631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.345727 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.345772 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.346076 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.346137 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:07:04.346467 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.338154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43689 (local address 127.0.98.2:43743)
0927 21:07:04.338312 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.338350 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.338806 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.339007 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.339010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.339595 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:04.339772 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.339776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.340410 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.340411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.341177 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.341179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.341490 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.341490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.341582 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.341968 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:04.341975 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:04.343274 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:04.344946 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:04.344952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:04.344954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:04.345240 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:04.345428 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:04.345428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:04.345429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:04.345567 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:04.345787 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:04.345792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:04.345945 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.346049 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.346214 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260927 21:07:04.346887 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43689
I20260927 21:07:04.346990 1287 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:04.347205 1287 heartbeater.cc:511] Master 127.0.98.62:43689 requested a full tablet report, sending...
I20260927 21:07:04.347536 962 ts_manager.cc:194] Registered new tserver with Master: 84be5e513bd346409cb9508f91a97f3c (127.0.98.2:44213)
I20260927 21:07:04.347898 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43743
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:04.367443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:04.446450 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:04.446662 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:04.446686 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:04.447818 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:04.447856 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:04.447882 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:04.449261 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:04.449445 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:04.449636 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:04.455044 1292 server_base.cc:1034] running on GCE node
W20260927 21:07:04.455013 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:04.455003 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:04.455013 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:04.455461 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:04.455706 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:04.456853 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790543224456843 us; error 40 us; skew 500 ppm
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:04.459515 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:04.459923 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42175/ using document root <none> and password file <none>
I20260927 21:07:04.460148 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:04.460211 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:04.460364 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:04.461462 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "80c59067ea0e4612915b675e75dc32b4"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "74876ca30733323cdf1b0a72b5b090f0"
server_key_iv: "0c2bf84e168dd220b481806e896183d0"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.461805 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80c59067ea0e4612915b675e75dc32b4"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "74876ca30733323cdf1b0a72b5b090f0"
server_key_iv: "0c2bf84e168dd220b481806e896183d0"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.463053 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:07:04.463802 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.463967 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:04.464053 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "80c59067ea0e4612915b675e75dc32b4"
format_stamp: "Formatted at 2026-09-27 21:07:04 on dist-test-slave-8bgs"
server_key: "74876ca30733323cdf1b0a72b5b090f0"
server_key_iv: "0c2bf84e168dd220b481806e896183d0"
server_key_version: "encryptionkey@0"
I20260927 21:07:04.464105 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:04.475235 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:04.477274 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:04.477432 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:04.477638 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:04.477926 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:04.477975 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.478017 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:04.478042 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:04.483625 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41275
I20260927 21:07:04.483711 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41275 every 8 connection(s)
I20260927 21:07:04.484014 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:04.491496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1292
I20260927 21:07:04.492406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:04.492583 392 external_mini_cluster.cc:1428] Setting key 5ead46892d191816f53120589f9abada
I20260927 21:07:04.492915 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.484706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57325 (local address 127.0.98.62:43689)
0927 21:07:04.484763 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.484765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.484968 (+ 203us) server_negotiation.cc:407] Connection header received
0927 21:07:04.485481 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.485483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.485495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.485550 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.486474 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.486987 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.488021 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.488215 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.488307 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.490267 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.490268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.490269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.490276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.491551 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.491951 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.491952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.491953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.492010 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.492238 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.492239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.492240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.492355 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.492429 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.492743 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.492820 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:07:04.493126 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.484694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43689 (local address 127.0.98.3:57325)
0927 21:07:04.484855 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.484889 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.485370 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.485631 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.485634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.486191 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:04.486380 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.486384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.487110 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.487111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.487920 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.487922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.488314 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.488315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.488444 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.488883 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:04.488891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:04.490165 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:04.491646 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:04.491652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:04.491654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:04.491876 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:04.492072 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:04.492073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:04.492074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:04.492174 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:04.492420 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:04.492426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:04.492580 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.492716 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.492882 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260927 21:07:04.493549 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43689
I20260927 21:07:04.493672 1421 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:04.493845 1421 heartbeater.cc:511] Master 127.0.98.62:43689 requested a full tablet report, sending...
I20260927 21:07:04.494271 961 ts_manager.cc:194] Registered new tserver with Master: 80c59067ea0e4612915b675e75dc32b4 (127.0.98.3:41275)
I20260927 21:07:04.494628 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57325
I20260927 21:07:04.504468 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:07:04.511880 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.506022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.0.98.62:43689)
0927 21:07:04.506075 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.506076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.506154 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:07:04.506206 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.506207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.506215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.506272 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.506826 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.507335 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.508099 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.508273 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.508321 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.509518 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.509519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.509519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.509524 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.510751 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.511048 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.511049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.511049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.511132 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.511301 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.511302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.511302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.511488 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.511579 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.511658 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.511740 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:07:04.514005 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:04.515712 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:04.522163 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.518165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.98.2:44213)
0927 21:07:04.518311 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.518316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.518326 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:04.518365 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.518367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.518383 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.518462 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:04.518736 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.519888 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521110 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.521800 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.521937 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.522068 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.522103 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260927 21:07:04.522177 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.517811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44213 (local address 127.0.0.1:56370)
0927 21:07:04.517920 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.517931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.517997 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.518483 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.518486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.518489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:04.518580 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.518582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.519993 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.519994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.521041 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.521042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.521886 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.521963 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.521977 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.522050 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.522102 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":76,"threads_started":1}
I20260927 21:07:04.522580 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.517987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35973 (local address 127.0.0.1:34762)
0927 21:07:04.518255 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.518266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.518316 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.518620 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.518621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.518624 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:04.518651 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.518653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.519678 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.519680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.520944 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.520947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.522367 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.522368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.522422 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.522426 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.522481 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.522517 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":63,"threads_started":1}
I20260927 21:07:04.522810 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.518250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.0.98.1:35973)
0927 21:07:04.518413 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.518417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.518425 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:04.518463 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.518465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.518479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.518546 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:04.518724 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.519581 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521020 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.521800 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.521938 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.522681 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.522727 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260927 21:07:04.522996 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.517786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41275 (local address 127.0.0.1:54580)
0927 21:07:04.517890 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.517930 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.517996 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.518578 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.518579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.518581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:04.518635 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.518637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.521121 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.521816 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.521818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.522797 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.522798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.522863 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.522867 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.522917 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.522955 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":57,"threads_started":1}
I20260927 21:07:04.523012 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff2c7f0e1a44540a3ade82daf45b740 (DEFAULT_TABLE table=test-table [id=60120484626f4c698c5cc8acf5592a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:04.523268 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.517951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.0.98.3:41275)
0927 21:07:04.518432 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.518438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.518447 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:04.518484 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.518486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.518501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.518579 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:04.518772 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.519889 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.521918 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.522642 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.522705 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.522756 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.523126 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.523171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":101,"threads_started":1}
I20260927 21:07:04.523440 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff2c7f0e1a44540a3ade82daf45b740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:04.523896 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff2c7f0e1a44540a3ade82daf45b740 (DEFAULT_TABLE table=test-table [id=60120484626f4c698c5cc8acf5592a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:04.524217 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff2c7f0e1a44540a3ade82daf45b740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:04.527308 1435 tablet_bootstrap.cc:492] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: Bootstrap starting.
I20260927 21:07:04.527755 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff2c7f0e1a44540a3ade82daf45b740 (DEFAULT_TABLE table=test-table [id=60120484626f4c698c5cc8acf5592a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:04.528069 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff2c7f0e1a44540a3ade82daf45b740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:04.528061 1436 tablet_bootstrap.cc:492] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: Bootstrap starting.
I20260927 21:07:04.528692 1436 tablet_bootstrap.cc:654] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:04.529241 1436 log.cc:824] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:04.529960 1436 tablet_bootstrap.cc:492] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: No bootstrap required, opened a new log
I20260927 21:07:04.530038 1436 ts_tablet_manager.cc:1397] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:04.531291 1438 tablet_bootstrap.cc:492] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: Bootstrap starting.
I20260927 21:07:04.531747 1435 tablet_bootstrap.cc:654] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:04.531926 1438 tablet_bootstrap.cc:654] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:04.531972 1436 raft_consensus.cc:348] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.532143 1436 raft_consensus.cc:374] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:04.532184 1436 raft_consensus.cc:729] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c59067ea0e4612915b675e75dc32b4, State: Initialized, Role: FOLLOWER
I20260927 21:07:04.532292 1436 consensus_queue.cc:260] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.532393 1435 log.cc:824] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:04.532490 1438 log.cc:824] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:04.532568 1436 ts_tablet_manager.cc:1428] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:07:04.532682 1421 heartbeater.cc:503] Master 127.0.98.62:43689 was elected leader, sending a full tablet report...
I20260927 21:07:04.533202 1438 tablet_bootstrap.cc:492] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: No bootstrap required, opened a new log
I20260927 21:07:04.533281 1438 ts_tablet_manager.cc:1397] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:04.533522 1435 tablet_bootstrap.cc:492] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: No bootstrap required, opened a new log
I20260927 21:07:04.533591 1435 ts_tablet_manager.cc:1397] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 21:07:04.534842 1438 raft_consensus.cc:348] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.535000 1438 raft_consensus.cc:374] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:04.535045 1438 raft_consensus.cc:729] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fa204385db4c62a9defe0786e09b52, State: Initialized, Role: FOLLOWER
I20260927 21:07:04.535187 1438 consensus_queue.cc:260] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.535468 1438 ts_tablet_manager.cc:1428] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:04.535494 1153 heartbeater.cc:503] Master 127.0.98.62:43689 was elected leader, sending a full tablet report...
I20260927 21:07:04.535596 1435 raft_consensus.cc:348] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.535748 1435 raft_consensus.cc:374] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:04.535817 1435 raft_consensus.cc:729] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84be5e513bd346409cb9508f91a97f3c, State: Initialized, Role: FOLLOWER
I20260927 21:07:04.535936 1435 consensus_queue.cc:260] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.536208 1435 ts_tablet_manager.cc:1428] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:07:04.536254 1287 heartbeater.cc:503] Master 127.0.98.62:43689 was elected leader, sending a full tablet report...
I20260927 21:07:04.565995 1442 raft_consensus.cc:482] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:04.566129 1442 raft_consensus.cc:504] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.566547 1442 leader_election.cc:288] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c59067ea0e4612915b675e75dc32b4 (127.0.98.3:41275), 84be5e513bd346409cb9508f91a97f3c (127.0.98.2:44213)
I20260927 21:07:04.569475 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.566711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44213 (local address 127.0.98.1:37589)
0927 21:07:04.566763 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.566776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.566818 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.566989 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.566990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.566992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:04.567064 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.567066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.567937 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.567938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.568669 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.568670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.569200 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.569201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.569283 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.569285 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.569358 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.569397 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:07:04.569468 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.566814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37589 (local address 127.0.98.2:44213)
0927 21:07:04.566875 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.566876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.566883 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:04.566905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.566906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.566914 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.566966 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:04.567163 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.567861 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.568764 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.569123 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.569200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.569210 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.569367 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.569404 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:07:04.569681 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.566711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41275 (local address 127.0.98.1:40777)
0927 21:07:04.566919 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:04.566926 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:04.566987 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:04.567176 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:04.567177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:04.567179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:04.567244 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.567245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.568021 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.568022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.568870 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:04.568872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.569411 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.569412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:04.569483 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.569486 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:04.569546 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:07:04.569593 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":12,"thread_start_us":123,"threads_started":1}
I20260927 21:07:04.569825 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff2c7f0e1a44540a3ade82daf45b740" candidate_uuid: "02fa204385db4c62a9defe0786e09b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84be5e513bd346409cb9508f91a97f3c" is_pre_election: true
I20260927 21:07:04.569922 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.566766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40777 (local address 127.0.98.3:41275)
0927 21:07:04.566821 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.566823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.566994 (+ 171us) server_negotiation.cc:407] Connection header received
0927 21:07:04.567029 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.567030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.567040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.567137 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:04.567342 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.567930 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.568955 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.569335 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.569409 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.569428 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.569847 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.569880 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:07:04.569990 1241 raft_consensus.cc:2454] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fa204385db4c62a9defe0786e09b52 in term 0.
I20260927 21:07:04.570219 1043 leader_election.cc:302] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fa204385db4c62a9defe0786e09b52, 84be5e513bd346409cb9508f91a97f3c; no voters:
I20260927 21:07:04.570341 1442 raft_consensus.cc:2781] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:04.570393 1442 raft_consensus.cc:482] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:04.570350 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff2c7f0e1a44540a3ade82daf45b740" candidate_uuid: "02fa204385db4c62a9defe0786e09b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c59067ea0e4612915b675e75dc32b4" is_pre_election: true
I20260927 21:07:04.570416 1442 raft_consensus.cc:3037] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:04.570492 1375 raft_consensus.cc:2454] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fa204385db4c62a9defe0786e09b52 in term 0.
W20260927 21:07:04.587998 1288 tablet.cc:2369] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:04.647665 1442 raft_consensus.cc:504] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.647878 1442 leader_election.cc:288] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [CANDIDATE]: Term 1 election: Requested vote from peers 80c59067ea0e4612915b675e75dc32b4 (127.0.98.3:41275), 84be5e513bd346409cb9508f91a97f3c (127.0.98.2:44213)
I20260927 21:07:04.648102 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff2c7f0e1a44540a3ade82daf45b740" candidate_uuid: "02fa204385db4c62a9defe0786e09b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c59067ea0e4612915b675e75dc32b4"
I20260927 21:07:04.648169 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff2c7f0e1a44540a3ade82daf45b740" candidate_uuid: "02fa204385db4c62a9defe0786e09b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84be5e513bd346409cb9508f91a97f3c"
I20260927 21:07:04.648227 1375 raft_consensus.cc:3037] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:04.648248 1241 raft_consensus.cc:3037] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:04.649240 1375 raft_consensus.cc:2454] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fa204385db4c62a9defe0786e09b52 in term 1.
I20260927 21:07:04.649240 1241 raft_consensus.cc:2454] T 3ff2c7f0e1a44540a3ade82daf45b740 P 84be5e513bd346409cb9508f91a97f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fa204385db4c62a9defe0786e09b52 in term 1.
I20260927 21:07:04.649451 1043 leader_election.cc:302] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fa204385db4c62a9defe0786e09b52, 84be5e513bd346409cb9508f91a97f3c; no voters:
I20260927 21:07:04.649562 1442 raft_consensus.cc:2781] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:04.649731 1442 raft_consensus.cc:686] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [term 1 LEADER]: Becoming Leader. State: Replica: 02fa204385db4c62a9defe0786e09b52, State: Running, Role: LEADER
I20260927 21:07:04.649838 1442 consensus_queue.cc:237] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } }
I20260927 21:07:04.650735 961 catalog_manager.cc:5495] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fa204385db4c62a9defe0786e09b52 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02fa204385db4c62a9defe0786e09b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c59067ea0e4612915b675e75dc32b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fa204385db4c62a9defe0786e09b52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84be5e513bd346409cb9508f91a97f3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44213 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:04.674028 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.666492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.0.98.1:35973)
0927 21:07:04.666577 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.666579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.666687 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:07:04.666772 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.666773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.666785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.666832 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.667432 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.667984 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.668686 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.668893 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.668974 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.671305 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.671319 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.671320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.671327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.672717 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.673034 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.673036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.673036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.673121 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.673321 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.673329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.673330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.673546 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.673746 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.673838 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.673920 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 21:07:04.690182 1154 tablet.cc:2369] T 3ff2c7f0e1a44540a3ade82daf45b740 P 02fa204385db4c62a9defe0786e09b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:07:04.734539 1422 tablet.cc:2369] T 3ff2c7f0e1a44540a3ade82daf45b740 P 80c59067ea0e4612915b675e75dc32b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:04.752008 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.744887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.0.98.1:35973)
0927 21:07:04.744937 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.744939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.745024 (+ 85us) server_negotiation.cc:407] Connection header received
0927 21:07:04.745117 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.745118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.745127 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.745191 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.745771 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.746296 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.747045 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.747317 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.747403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.749629 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.749631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.749631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.749639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.750949 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.751362 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.751363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.751363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.751421 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.751577 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.751578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.751578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.751683 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.751767 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.751883 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.751944 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:07:04 dist-test-slave-8bgs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:04.848971 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:04.841842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.0.98.1:35973)
0927 21:07:04.841899 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:04.841901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:04.841990 (+ 89us) server_negotiation.cc:407] Connection header received
0927 21:07:04.842081 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:04.842082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:04.842094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:04.842159 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:04.842735 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.843277 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.844011 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:04.844213 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:04.844295 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:04.846684 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:04.846686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:04.846686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:04.846694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:04.847911 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.848265 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.848266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.848267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.848322 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:04.848505 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:04.848506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:04.848506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:04.848599 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:04.848750 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:04.848789 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 21:07:04.848864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 21:07:04.849406 1067 tablet_service.cc:2908] Rejecting scan request for tablet 3ff2c7f0e1a44540a3ade82daf45b740: Uninitialized: clean time has not yet been initialized
I20260927 21:07:04.850209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1024
I20260927 21:07:04.855574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1158
I20260927 21:07:04.859975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1292
I20260927 21:07:04.864292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 930
2026-09-27T21:07:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2110 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:07 dist-test-slave-8bgs krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:07 dist-test-slave-8bgs krb5kdc[1459](info): set up 2 sockets
Sep 27 21:07:07 dist-test-slave-8bgs krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260927 21:07:08.068261 392 mini_kdc.cc:121] Time spent starting KDC: real 3.182s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 21:07:08.210774 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.142s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 21:07:08.380172 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.169s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 21:07:08.596942 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.217s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/test-user.keytab.
W20260927 21:07:09.076540 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.479s user 0.000s sys 0.002s
Sep 27 21:07:09 dist-test-slave-8bgs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 21:07:10.221324 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 1.124s user 0.000s sys 0.002s
I20260927 21:07:10.221689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33351
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:10.301542 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:10.301746 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:10.301767 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:10.302856 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:10.302894 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:07:10.302907 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:10.302920 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:10.302932 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:10.302944 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:10.304399 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33351
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:10.304589 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:10.304750 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:10.309299 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:10.309305 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.309432 1472 server_base.cc:1034] running on GCE node
W20260927 21:07:10.309501 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.309788 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:10.310087 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:10.311230 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790543230311220 us; error 49 us; skew 500 ppm
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:10.313689 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:10.314121 1472 webserver.cc:460] Webserver started at http://127.0.98.62:45227/ using document root <none> and password file <none>
I20260927 21:07:10.314309 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:10.314350 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:10.314441 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:10.315523 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "66906d99ca834483b1f5055ad8e4770d"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "e288c7349e31a0162bf53de28a39f45b"
server_key_iv: "d6694b14cd91d18e4d93aab8ed8ec76e"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.315835 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "66906d99ca834483b1f5055ad8e4770d"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "e288c7349e31a0162bf53de28a39f45b"
server_key_iv: "d6694b14cd91d18e4d93aab8ed8ec76e"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.317140 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:10.317765 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.317950 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:10.318038 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "66906d99ca834483b1f5055ad8e4770d"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "e288c7349e31a0162bf53de28a39f45b"
server_key_iv: "d6694b14cd91d18e4d93aab8ed8ec76e"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.318092 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:10.349980 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:10.352272 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:10.352382 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:10.356966 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33351
I20260927 21:07:10.357010 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33351 every 8 connection(s)
I20260927 21:07:10.357329 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:10.357968 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.360219 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: Bootstrap starting.
I20260927 21:07:10.360811 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.361085 1550 log.cc:824] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:10.361686 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: No bootstrap required, opened a new log
I20260927 21:07:10.363761 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66906d99ca834483b1f5055ad8e4770d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260927 21:07:10.363909 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.363953 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66906d99ca834483b1f5055ad8e4770d, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.364110 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66906d99ca834483b1f5055ad8e4770d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260927 21:07:10.364187 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:10.364225 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:10.364277 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:10.364775 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66906d99ca834483b1f5055ad8e4770d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260927 21:07:10.364886 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66906d99ca834483b1f5055ad8e4770d; no voters:
I20260927 21:07:10.365056 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:10.365104 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:10.365264 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [term 1 LEADER]: Becoming Leader. State: Replica: 66906d99ca834483b1f5055ad8e4770d, State: Running, Role: LEADER
I20260927 21:07:10.365311 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:10.365366 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66906d99ca834483b1f5055ad8e4770d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } }
I20260927 21:07:10.365705 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66906d99ca834483b1f5055ad8e4770d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66906d99ca834483b1f5055ad8e4770d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } } }
I20260927 21:07:10.365746 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66906d99ca834483b1f5055ad8e4770d. Latest consensus state: current_term: 1 leader_uuid: "66906d99ca834483b1f5055ad8e4770d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66906d99ca834483b1f5055ad8e4770d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33351 } } }
I20260927 21:07:10.365887 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:10.366102 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:10.366272 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:10.366680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1472
I20260927 21:07:10.366775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:10.366942 392 external_mini_cluster.cc:1428] Setting key c8a2ed1eb41b8a3c01df17c8a013de71
I20260927 21:07:10.367300 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:10.369941 1562 catalog_manager.cc:1347] Generated new cluster ID: b50cc9c46bb24479a8b60b53c746cf75
I20260927 21:07:10.370007 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:10.375898 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:10.376606 1562 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:10.378186 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.367394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.0.98.62:33351)
0927 21:07:10.367665 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.367674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.367812 (+ 138us) server_negotiation.cc:407] Connection header received
0927 21:07:10.367949 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.367957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.368015 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.368161 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.368931 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.369839 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.370912 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.371879 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.371977 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.374759 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.374763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.374763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.374775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.376726 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.377090 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.377093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.377094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.377181 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.377384 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.377385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.377386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.377595 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.377775 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.377840 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.377996 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
I20260927 21:07:10.382997 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: Generated new TSK 0
I20260927 21:07:10.383179 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:10.409963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33351
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:10.510044 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:10.510263 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:10.510304 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:10.511488 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:10.511540 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:10.511576 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:10.511624 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:10.513036 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:10.513302 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:10.513514 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:10.519438 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.519505 1576 server_base.cc:1034] running on GCE node
W20260927 21:07:10.519436 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:10.519455 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.519903 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:10.520159 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:10.521353 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790543230521337 us; error 27 us; skew 500 ppm
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:10.524324 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:10.524827 1576 webserver.cc:460] Webserver started at http://127.0.98.1:42311/ using document root <none> and password file <none>
I20260927 21:07:10.525056 1576 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:10.525100 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:10.525233 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:10.526221 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "59ac76ad17a7412eb751079efe879b43"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "820b08b67c324c77549dde3bf003c463"
server_key_iv: "979e841155c8a66ec68e999a9efc912f"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.526578 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59ac76ad17a7412eb751079efe879b43"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "820b08b67c324c77549dde3bf003c463"
server_key_iv: "979e841155c8a66ec68e999a9efc912f"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.527971 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:10.528712 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.528861 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:10.528956 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "59ac76ad17a7412eb751079efe879b43"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "820b08b67c324c77549dde3bf003c463"
server_key_iv: "979e841155c8a66ec68e999a9efc912f"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.529024 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:10.555590 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:10.558176 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:10.558357 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:10.558862 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:10.558899 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.558926 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:10.558976 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:10.566452 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41627
I20260927 21:07:10.566525 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41627 every 8 connection(s)
I20260927 21:07:10.566932 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:10.568867 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.559390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54697 (local address 127.0.98.62:33351)
0927 21:07:10.559460 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.559462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.559701 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:07:10.560447 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.560448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.560457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.560500 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.561406 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.562160 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.563190 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.563405 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.563473 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.566065 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.566066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.566066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.566072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.567714 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.568089 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.568091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.568091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.568153 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.568371 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.568371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.568371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.568478 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.568535 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.568729 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.568799 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:07:10.569046 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.559367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.98.1:54697)
0927 21:07:10.559552 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.559603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.560329 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.560577 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.560581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.560966 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:10.561293 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.561300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.562247 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.562248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.563050 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.563052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.563503 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.563504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.563617 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.564092 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:10.564100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:10.565970 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:10.567783 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.567787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.567789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.567991 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.568218 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.568218 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.568219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.568313 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.568527 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:10.568529 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:10.568643 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.568715 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.568828 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260927 21:07:10.569540 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33351
I20260927 21:07:10.569613 1707 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:10.569785 1707 heartbeater.cc:511] Master 127.0.98.62:33351 requested a full tablet report, sending...
I20260927 21:07:10.570261 1504 ts_manager.cc:194] Registered new tserver with Master: 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627)
I20260927 21:07:10.571054 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54697
I20260927 21:07:10.575394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1576
I20260927 21:07:10.575500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:10.575671 392 external_mini_cluster.cc:1428] Setting key a821229c5618665d7eb7f411da29ee49
I20260927 21:07:10.576280 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.570524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.98.1:57935)
0927 21:07:10.570657 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.570670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.570713 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.571454 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.571455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.571624 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:10.571688 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.571689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.572311 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.572312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.572831 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.572832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.573202 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.573203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.573269 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.573628 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:10.573632 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:10.573880 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:10.575188 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.575189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.575190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.575355 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.575585 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.575586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.575586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.575648 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.575968 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:10.575969 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:10.576017 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.576091 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.576174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260927 21:07:10.576282 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.570578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57935 (local address 127.0.98.62:33351)
0927 21:07:10.571346 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.571348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.571357 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:10.571377 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.571378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.571387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.571434 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.571772 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.572224 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.572912 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.573118 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.573200 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.573985 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.573986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.573987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.573993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.575108 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.575438 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.575440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.575440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.575492 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.575723 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.575724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.575725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.575873 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.575961 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.576129 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.576195 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:10.599562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33351
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:10.681742 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:10.681972 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:10.682013 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:10.683245 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:10.683305 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:10.683336 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:10.683377 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:10.684877 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:10.685129 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:10.685369 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:10.690219 1716 server_base.cc:1034] running on GCE node
W20260927 21:07:10.690085 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:10.690160 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:10.690084 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.690570 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:10.690819 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:10.691951 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790543230691940 us; error 27 us; skew 500 ppm
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:10.694269 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:10.694625 1716 webserver.cc:460] Webserver started at http://127.0.98.2:42081/ using document root <none> and password file <none>
I20260927 21:07:10.694806 1716 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:10.694846 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:10.694929 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:10.695935 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "c2bf21e456b44c198a5003c48d464a4f"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "e63eff33a984cf9d7ea4adcdffd3e0ec"
server_key_iv: "cf12b25b173ac02965e660717064456d"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.696223 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c2bf21e456b44c198a5003c48d464a4f"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "e63eff33a984cf9d7ea4adcdffd3e0ec"
server_key_iv: "cf12b25b173ac02965e660717064456d"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.697429 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:10.698071 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.698239 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:10.698335 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "c2bf21e456b44c198a5003c48d464a4f"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "e63eff33a984cf9d7ea4adcdffd3e0ec"
server_key_iv: "cf12b25b173ac02965e660717064456d"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.698410 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:10.729903 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:10.732227 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:10.732389 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:10.732919 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:10.732980 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.733023 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:10.733050 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:10.739876 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36109
I20260927 21:07:10.740196 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36109 every 8 connection(s)
I20260927 21:07:10.740281 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:10.742442 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.733401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50297 (local address 127.0.98.62:33351)
0927 21:07:10.733473 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.733475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.733635 (+ 160us) server_negotiation.cc:407] Connection header received
0927 21:07:10.734192 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.734194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.734211 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.734288 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.735213 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.735745 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.736749 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.737011 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.737092 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.739913 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.739914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.739929 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.739938 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.741287 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.741762 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.741763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.741764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.741826 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.742007 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.742008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.742008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.742097 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.742141 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.742318 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.742384 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:07:10.742655 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.733371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.98.2:50297)
0927 21:07:10.733511 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.733551 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.734098 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.734343 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.734347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.734801 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:10.735096 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.735104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.735857 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.735858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.736657 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.736660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.737112 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.737113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.737236 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.737690 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:10.737699 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:10.739801 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:10.741352 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.741356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.741358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.741697 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.741887 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.741887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.741888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.741971 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.742131 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:10.742134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:10.742231 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.742305 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.742423 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260927 21:07:10.743198 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33351
I20260927 21:07:10.743275 1847 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:10.743456 1847 heartbeater.cc:511] Master 127.0.98.62:33351 requested a full tablet report, sending...
I20260927 21:07:10.743808 1504 ts_manager.cc:194] Registered new tserver with Master: c2bf21e456b44c198a5003c48d464a4f (127.0.98.2:36109)
I20260927 21:07:10.744165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1716
I20260927 21:07:10.744208 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50297
I20260927 21:07:10.744277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:10.744470 392 external_mini_cluster.cc:1428] Setting key cc14d51983aee5b7548e87e7d5f9cac6
I20260927 21:07:10.749812 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.744302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35419 (local address 127.0.98.62:33351)
0927 21:07:10.744393 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.744395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.744408 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:07:10.744467 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.744467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.744476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.744536 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.744867 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.745341 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.746078 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.746390 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.746461 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.747445 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.747446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.747447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.747452 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.748697 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.749095 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.749096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.749097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.749144 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.749349 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.749349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.749349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.749488 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.749540 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.749699 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.749753 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:07:10.749926 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.744231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.98.2:35419)
0927 21:07:10.744360 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.744371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.744417 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.744542 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.744543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.744725 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:10.744791 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.744793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.745424 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.745425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.745999 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.746001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.746464 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.746464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.746510 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.746924 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:10.746930 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:10.747356 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:10.748777 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.748778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.748778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.749006 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.749205 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.749206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.749206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.749283 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.749554 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:10.749554 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:10.749600 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.749682 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.749795 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:10.769222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33351
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:10.849229 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:10.849424 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:10.849447 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:10.850597 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:10.850638 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:10.850685 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:10.850720 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:10.852210 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:10.852454 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:10.852684 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:10.857641 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:10.857596 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.857645 1856 server_base.cc:1034] running on GCE node
W20260927 21:07:10.857616 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:10.858013 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:10.858209 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:10.859338 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790543230859325 us; error 28 us; skew 500 ppm
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:10.861696 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:10.862150 1856 webserver.cc:460] Webserver started at http://127.0.98.3:46371/ using document root <none> and password file <none>
I20260927 21:07:10.862382 1856 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:10.862426 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:10.862591 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:10.863696 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "54e14d8196994c758d661b7a533c9024"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "aec86fced59900841da859db4ef44022"
server_key_iv: "60c60730120d4d081d4eeb1db3e2afd8"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.864050 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54e14d8196994c758d661b7a533c9024"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "aec86fced59900841da859db4ef44022"
server_key_iv: "60c60730120d4d081d4eeb1db3e2afd8"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.865396 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:10.866124 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.866303 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:10.866436 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "54e14d8196994c758d661b7a533c9024"
format_stamp: "Formatted at 2026-09-27 21:07:10 on dist-test-slave-8bgs"
server_key: "aec86fced59900841da859db4ef44022"
server_key_iv: "60c60730120d4d081d4eeb1db3e2afd8"
server_key_version: "encryptionkey@0"
I20260927 21:07:10.866506 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:10.888042 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:10.889743 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:10.889847 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:10.890267 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:10.890301 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.890327 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:10.890379 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:10.896906 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42231
I20260927 21:07:10.896958 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42231 every 8 connection(s)
I20260927 21:07:10.897248 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:10.900115 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.890748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48423 (local address 127.0.98.62:33351)
0927 21:07:10.890804 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.890806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.890978 (+ 172us) server_negotiation.cc:407] Connection header received
0927 21:07:10.891607 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.891609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.891625 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.891702 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.892547 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.893378 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.894948 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.895274 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.895351 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.897800 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.897801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.897801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.897807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.898932 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.899363 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.899365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.899365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.899424 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.899624 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.899625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.899625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.899720 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.899767 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.899971 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.900052 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:07:10.900265 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.890723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.98.3:48423)
0927 21:07:10.890874 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.890910 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.891473 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.891792 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.891796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.892161 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:10.892443 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.892449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.893492 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.893494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.894872 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.894876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.895365 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.895366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.895461 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.896036 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:10.896046 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:10.897713 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:10.898997 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.899002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.899004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.899278 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.899492 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.899492 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.899493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.899584 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.899756 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:10.899759 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:10.899860 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.899928 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.900064 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260927 21:07:10.900708 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33351
I20260927 21:07:10.900853 1987 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:10.901042 1987 heartbeater.cc:511] Master 127.0.98.62:33351 requested a full tablet report, sending...
I20260927 21:07:10.901453 1503 ts_manager.cc:194] Registered new tserver with Master: 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231)
I20260927 21:07:10.901844 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48423
I20260927 21:07:10.903798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 1856
I20260927 21:07:10.903874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:10.904059 392 external_mini_cluster.cc:1428] Setting key 84e245e4ffb32aae378273f164de6a08
I20260927 21:07:10.904861 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:07:10.907878 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.901782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54319 (local address 127.0.98.62:33351)
0927 21:07:10.901820 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.901822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.901952 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:07:10.902080 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.902081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.902092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.902186 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.902612 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.903333 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.904108 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.904390 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.904446 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.905365 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.905367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.905367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.905375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.906837 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.907239 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.907240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.907240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.907288 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.907479 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.907479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.907480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.907569 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.907621 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.907772 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.907823 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:10.908028 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.901727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.98.3:54319)
0927 21:07:10.901869 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.901879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.901974 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.902196 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.902197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.902419 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:10.902503 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.902505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.903415 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.903415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.904013 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.904015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.904498 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.904499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.904569 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.905012 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:10.905017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:10.905284 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:10.906918 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.906919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.906919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.907170 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.907350 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:10.907351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:10.907352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:10.907421 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:10.907630 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:10.907631 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:10.907677 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.907758 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.907872 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:07:10 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:10.936997 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.929863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.98.62:33351)
0927 21:07:10.929908 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.929910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.929919 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:10.929950 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.929954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.929964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.930024 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:10.930881 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.931564 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.932462 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.932643 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.932698 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.934795 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:10.934797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:10.934797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:10.934803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:10.936076 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.936396 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.936397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.936397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.936447 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:10.936611 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:10.936612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:10.936612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:10.936708 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:10.936780 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.936871 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.936931 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:10.938714 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:10.940366 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:10.946884 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.942665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41627 (local address 127.0.0.1:40072)
0927 21:07:10.943011 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.943020 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.943065 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.943705 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.943706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.943707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:10.943743 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.943744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.944733 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.944734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.945715 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.945717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.946589 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.946590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.946663 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.946677 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.946755 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.946801 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":81,"threads_started":1}
I20260927 21:07:10.946940 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.942771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.0.98.1:41627)
0927 21:07:10.943478 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.943483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.943492 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:10.943544 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.943546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.943561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.943639 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:10.943828 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.944651 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.945822 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.946493 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.946571 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.946642 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.946780 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.946821 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":76,"threads_started":1}
I20260927 21:07:10.947186 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.942851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.0.1:41198)
0927 21:07:10.943299 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.943308 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.943359 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.943592 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.943593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.943594 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:10.943647 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.943648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.944798 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.944799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.946129 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.946131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.946944 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.946945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.947019 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.947025 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.947057 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.947115 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":52,"threads_started":1}
I20260927 21:07:10.947302 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.942633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42231 (local address 127.0.0.1:39538)
0927 21:07:10.942787 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.942829 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.942897 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.943244 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.943247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.943249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:10.943316 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.943318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.944625 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.944626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.946006 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.946008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.947005 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.947006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.947080 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.947132 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.947198 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.947239 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":72,"thread_start_us":75,"threads_started":1}
I20260927 21:07:10.947485 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.942833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.98.3:42231)
0927 21:07:10.943033 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.943038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.943050 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:07:10.943123 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.943126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.943143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.943247 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:10.943403 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.944544 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.946086 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.946828 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.946896 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.946967 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.947355 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.947385 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260927 21:07:10.947515 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.943033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.0.98.2:36109)
0927 21:07:10.943220 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.943224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.943353 (+ 129us) server_negotiation.cc:407] Connection header received
0927 21:07:10.943483 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.943485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.943501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.943593 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:10.943704 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.944737 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.946229 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.946851 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.947247 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.947334 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.947364 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.947402 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260927 21:07:10.948148 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3cb97d33564f5cbbe54426c80eee29 (DEFAULT_TABLE table=test-table [id=3651d57a925b4fff8a4d0f7d157bfbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:10.948303 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3cb97d33564f5cbbe54426c80eee29 (DEFAULT_TABLE table=test-table [id=3651d57a925b4fff8a4d0f7d157bfbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:10.948361 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3cb97d33564f5cbbe54426c80eee29 (DEFAULT_TABLE table=test-table [id=3651d57a925b4fff8a4d0f7d157bfbf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:10.948501 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3cb97d33564f5cbbe54426c80eee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.948658 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3cb97d33564f5cbbe54426c80eee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.948686 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3cb97d33564f5cbbe54426c80eee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.951599 2010 tablet_bootstrap.cc:492] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Bootstrap starting.
I20260927 21:07:10.952234 2010 tablet_bootstrap.cc:654] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.952358 2012 tablet_bootstrap.cc:492] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Bootstrap starting.
I20260927 21:07:10.952831 2010 log.cc:824] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:10.952962 2012 tablet_bootstrap.cc:654] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.953281 2012 log.cc:824] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:10.953578 2010 tablet_bootstrap.cc:492] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: No bootstrap required, opened a new log
I20260927 21:07:10.953600 2011 tablet_bootstrap.cc:492] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Bootstrap starting.
I20260927 21:07:10.953652 2010 ts_tablet_manager.cc:1397] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:10.953992 2012 tablet_bootstrap.cc:492] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: No bootstrap required, opened a new log
I20260927 21:07:10.954059 2012 ts_tablet_manager.cc:1397] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:10.954195 2011 tablet_bootstrap.cc:654] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.954526 2011 log.cc:824] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:10.955313 2011 tablet_bootstrap.cc:492] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: No bootstrap required, opened a new log
I20260927 21:07:10.955381 2011 ts_tablet_manager.cc:1397] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:10.955773 2012 raft_consensus.cc:348] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.955919 2012 raft_consensus.cc:374] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.955958 2012 raft_consensus.cc:729] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bf21e456b44c198a5003c48d464a4f, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.956073 2012 consensus_queue.cc:260] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.956023 2010 raft_consensus.cc:348] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.956228 2010 raft_consensus.cc:374] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.956262 2010 raft_consensus.cc:729] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e14d8196994c758d661b7a533c9024, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.956321 2012 ts_tablet_manager.cc:1428] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:07:10.956382 2010 consensus_queue.cc:260] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.956432 1847 heartbeater.cc:503] Master 127.0.98.62:33351 was elected leader, sending a full tablet report...
I20260927 21:07:10.956647 2010 ts_tablet_manager.cc:1428] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:07:10.956670 1987 heartbeater.cc:503] Master 127.0.98.62:33351 was elected leader, sending a full tablet report...
I20260927 21:07:10.957525 2011 raft_consensus.cc:348] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.957691 2011 raft_consensus.cc:374] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.957738 2011 raft_consensus.cc:729] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ac76ad17a7412eb751079efe879b43, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.957852 2011 consensus_queue.cc:260] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.958101 2011 ts_tablet_manager.cc:1428] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:07:10.958170 1707 heartbeater.cc:503] Master 127.0.98.62:33351 was elected leader, sending a full tablet report...
I20260927 21:07:10.959376 2016 raft_consensus.cc:482] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:10.959450 2016 raft_consensus.cc:504] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.959661 2016 leader_election.cc:288] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231), 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627)
I20260927 21:07:10.963264 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.959901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48839 (local address 127.0.98.1:41627)
0927 21:07:10.959937 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.959939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.959947 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:10.959999 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.960000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.960012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.960066 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:10.960230 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.960973 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.962383 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.962914 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.962985 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.962994 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.963173 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.963204 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:10.963266 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.959895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42335 (local address 127.0.98.3:42231)
0927 21:07:10.959933 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.959935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.960090 (+ 155us) server_negotiation.cc:407] Connection header received
0927 21:07:10.960172 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.960173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.960180 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.960262 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:10.960390 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.961042 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.962383 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.962915 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.962985 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.962994 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.963173 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.963204 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:10.963305 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.959835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41627 (local address 127.0.98.2:48839)
0927 21:07:10.959887 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.959899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.959938 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.960086 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.960087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.960089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:10.960157 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.960158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.961042 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.961042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.962300 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.962303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.962994 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.962995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.963079 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.963082 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.963167 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.963216 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":71}
I20260927 21:07:10.963382 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.959835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42231 (local address 127.0.98.2:42335)
0927 21:07:10.960005 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:10.960022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:10.960113 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:10.960275 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:10.960276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:10.960277 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:10.960331 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.960332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.961105 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.961105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.962300 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:10.962303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.962994 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.962995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:10.963079 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.963083 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:10.963167 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:07:10.963216 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":20,"thread_start_us":70,"threads_started":1}
I20260927 21:07:10.963624 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3cb97d33564f5cbbe54426c80eee29" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ac76ad17a7412eb751079efe879b43" is_pre_election: true
I20260927 21:07:10.963729 1661 raft_consensus.cc:2454] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 0.
I20260927 21:07:10.963694 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3cb97d33564f5cbbe54426c80eee29" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e14d8196994c758d661b7a533c9024" is_pre_election: true
I20260927 21:07:10.963815 1941 raft_consensus.cc:2454] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 0.
I20260927 21:07:10.963909 1733 leader_election.cc:302] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ac76ad17a7412eb751079efe879b43, c2bf21e456b44c198a5003c48d464a4f; no voters:
I20260927 21:07:10.964025 2016 raft_consensus.cc:2781] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:10.964087 2016 raft_consensus.cc:482] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:10.964119 2016 raft_consensus.cc:3037] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:10.964939 2016 raft_consensus.cc:504] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.965094 2016 leader_election.cc:288] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 election: Requested vote from peers 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231), 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627)
I20260927 21:07:10.965224 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3cb97d33564f5cbbe54426c80eee29" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ac76ad17a7412eb751079efe879b43"
I20260927 21:07:10.965314 1661 raft_consensus.cc:3037] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:10.966001 1661 raft_consensus.cc:2454] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 1.
I20260927 21:07:10.966147 1733 leader_election.cc:302] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ac76ad17a7412eb751079efe879b43, c2bf21e456b44c198a5003c48d464a4f; no voters:
I20260927 21:07:10.966156 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3cb97d33564f5cbbe54426c80eee29" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e14d8196994c758d661b7a533c9024"
I20260927 21:07:10.966224 1941 raft_consensus.cc:3037] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:10.966262 2016 raft_consensus.cc:2781] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:10.966390 2016 raft_consensus.cc:686] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 1 LEADER]: Becoming Leader. State: Replica: c2bf21e456b44c198a5003c48d464a4f, State: Running, Role: LEADER
I20260927 21:07:10.966490 2016 consensus_queue.cc:237] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } }
I20260927 21:07:10.967060 1941 raft_consensus.cc:2454] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 1.
I20260927 21:07:10.967172 1503 catalog_manager.cc:5495] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f reported cstate change: term changed from 0 to 1, leader changed from <none> to c2bf21e456b44c198a5003c48d464a4f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } health_report { overall_health: HEALTHY } } }
I20260927 21:07:10.974327 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:10.972323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.0.98.2:36109)
0927 21:07:10.972358 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:10.972361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:10.972392 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:07:10.972478 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:10.972479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:10.972492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:10.972540 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:10.972639 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.973077 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.973769 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:10.973932 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:10.973977 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:10.974131 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:10.974205 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:10.974255 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:10.974270 (+ 15us) server_negotiation.cc:299] Negotiation successful
0927 21:07:10.974284 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:10.975840 1661 raft_consensus.cc:1261] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Refusing update from remote peer c2bf21e456b44c198a5003c48d464a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:10.975876 1941 raft_consensus.cc:1261] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Refusing update from remote peer c2bf21e456b44c198a5003c48d464a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:10.976140 2020 consensus_queue.cc:1035] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:10.976264 2016 consensus_queue.cc:1035] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:10.978288 2023 mvcc.cc:205] Tried to move back new op lower bound from 7334065074074841088 to 7334065074039320576. Current Snapshot: MvccSnapshot[applied={T|T < 7334065074074841088}]
I20260927 21:07:10.978716 2024 mvcc.cc:205] Tried to move back new op lower bound from 7334065074074841088 to 7334065074039320576. Current Snapshot: MvccSnapshot[applied={T|T < 7334065074074841088}]
I20260927 21:07:10.981647 1502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:10.981717 1502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:10.982410 1502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: Sending DeleteTablet for 3 replicas of tablet 3b3cb97d33564f5cbbe54426c80eee29
I20260927 21:07:10.982776 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3cb97d33564f5cbbe54426c80eee29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39538
I20260927 21:07:10.982781 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3cb97d33564f5cbbe54426c80eee29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41198
I20260927 21:07:10.982970 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3cb97d33564f5cbbe54426c80eee29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40072
I20260927 21:07:10.983055 2034 tablet_replica.cc:330] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: stopping tablet replica
I20260927 21:07:10.983171 2034 raft_consensus.cc:2229] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:10.983178 2033 tablet_replica.cc:330] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: stopping tablet replica
I20260927 21:07:10.983134 1502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:10.983263 1502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:10.983259 2033 raft_consensus.cc:2229] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:10.983271 2034 raft_consensus.cc:2258] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:10.983713 2034 ts_tablet_manager.cc:1905] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:10.984409 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 94cb963477c540509e930389db88ebf9 (DEFAULT_TABLE table=test-table [id=e0df9927df644b7690a8cda12d6520ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:10.984436 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 94cb963477c540509e930389db88ebf9 (DEFAULT_TABLE table=test-table [id=e0df9927df644b7690a8cda12d6520ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:10.984436 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 94cb963477c540509e930389db88ebf9 (DEFAULT_TABLE table=test-table [id=e0df9927df644b7690a8cda12d6520ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:10.984524 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cb963477c540509e930389db88ebf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.984531 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cb963477c540509e930389db88ebf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.984531 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94cb963477c540509e930389db88ebf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:10.985723 2011 tablet_bootstrap.cc:492] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Bootstrap starting.
I20260927 21:07:10.985754 2012 tablet_bootstrap.cc:492] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Bootstrap starting.
I20260927 21:07:10.985783 2010 tablet_bootstrap.cc:492] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Bootstrap starting.
I20260927 21:07:10.985940 2035 tablet_replica.cc:330] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: stopping tablet replica
I20260927 21:07:10.986032 2035 raft_consensus.cc:2229] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:10.986128 2035 raft_consensus.cc:2258] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:10.986159 2012 tablet_bootstrap.cc:654] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.986209 2010 tablet_bootstrap.cc:654] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.986495 2033 raft_consensus.cc:2258] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:10.986856 2011 tablet_bootstrap.cc:654] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:10.986968 2035 ts_tablet_manager.cc:1905] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:10.988345 2034 ts_tablet_manager.cc:1918] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:10.988404 2035 ts_tablet_manager.cc:1918] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:10.988430 2034 log.cc:1196] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/3b3cb97d33564f5cbbe54426c80eee29
I20260927 21:07:10.988476 2035 log.cc:1196] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/3b3cb97d33564f5cbbe54426c80eee29
I20260927 21:07:10.988690 2034 ts_tablet_manager.cc:1939] T 3b3cb97d33564f5cbbe54426c80eee29 P c2bf21e456b44c198a5003c48d464a4f: Deleting consensus metadata
I20260927 21:07:10.988739 2035 ts_tablet_manager.cc:1939] T 3b3cb97d33564f5cbbe54426c80eee29 P 59ac76ad17a7412eb751079efe879b43: Deleting consensus metadata
I20260927 21:07:10.989081 1489 catalog_manager.cc:4841] TS 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627): tablet 3b3cb97d33564f5cbbe54426c80eee29 (table test-table [id=3651d57a925b4fff8a4d0f7d157bfbf9]) successfully deleted
I20260927 21:07:10.989301 1489 catalog_manager.cc:4841] TS c2bf21e456b44c198a5003c48d464a4f (127.0.98.2:36109): tablet 3b3cb97d33564f5cbbe54426c80eee29 (table test-table [id=3651d57a925b4fff8a4d0f7d157bfbf9]) successfully deleted
I20260927 21:07:10.989444 2033 ts_tablet_manager.cc:1905] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:10.990851 2033 ts_tablet_manager.cc:1918] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:10.990926 2033 log.cc:1196] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/3b3cb97d33564f5cbbe54426c80eee29
I20260927 21:07:10.991206 2033 ts_tablet_manager.cc:1939] T 3b3cb97d33564f5cbbe54426c80eee29 P 54e14d8196994c758d661b7a533c9024: Deleting consensus metadata
I20260927 21:07:10.991503 2010 tablet_bootstrap.cc:492] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: No bootstrap required, opened a new log
I20260927 21:07:10.991560 2010 ts_tablet_manager.cc:1397] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 21:07:10.991611 1488 catalog_manager.cc:4841] TS 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231): tablet 3b3cb97d33564f5cbbe54426c80eee29 (table test-table [id=3651d57a925b4fff8a4d0f7d157bfbf9]) successfully deleted
I20260927 21:07:10.991707 2010 raft_consensus.cc:348] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:10.991868 2012 tablet_bootstrap.cc:492] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: No bootstrap required, opened a new log
I20260927 21:07:10.991912 2012 ts_tablet_manager.cc:1397] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 21:07:10.992053 2012 raft_consensus.cc:348] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:10.992117 2012 raft_consensus.cc:374] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.992143 2012 raft_consensus.cc:729] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bf21e456b44c198a5003c48d464a4f, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.991798 2010 raft_consensus.cc:374] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.992187 2012 consensus_queue.cc:260] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:10.992219 2010 raft_consensus.cc:729] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e14d8196994c758d661b7a533c9024, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.992280 2012 ts_tablet_manager.cc:1428] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:10.992272 2010 consensus_queue.cc:260] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:10.992372 2010 ts_tablet_manager.cc:1428] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:10.992461 2011 tablet_bootstrap.cc:492] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: No bootstrap required, opened a new log
I20260927 21:07:10.992507 2011 ts_tablet_manager.cc:1397] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260927 21:07:10.992642 2011 raft_consensus.cc:348] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:10.992712 2011 raft_consensus.cc:374] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:10.992740 2011 raft_consensus.cc:729] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ac76ad17a7412eb751079efe879b43, State: Initialized, Role: FOLLOWER
I20260927 21:07:10.992779 2011 consensus_queue.cc:260] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:10.992861 2011 ts_tablet_manager.cc:1428] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:11.043764 2017 raft_consensus.cc:482] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:11.043875 2017 raft_consensus.cc:504] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.044164 2017 leader_election.cc:288] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627), c2bf21e456b44c198a5003c48d464a4f (127.0.98.2:36109)
I20260927 21:07:11.047277 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.044470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49907 (local address 127.0.98.1:41627)
0927 21:07:11.044510 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.044512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.044521 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:11.044570 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.044571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.044579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.044631 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:11.044826 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.045306 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.046315 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.046874 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.046945 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.046955 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.047170 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.047207 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:11.047295 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.044379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41627 (local address 127.0.98.3:49907)
0927 21:07:11.044449 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:11.044462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:11.044514 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:11.044679 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:11.044680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:11.044682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:11.044750 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.044752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.045390 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.045391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.046215 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.046218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.046961 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.046962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.047049 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.047053 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:11.047161 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:07:11.047206 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260927 21:07:11.047601 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cb963477c540509e930389db88ebf9" candidate_uuid: "54e14d8196994c758d661b7a533c9024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ac76ad17a7412eb751079efe879b43" is_pre_election: true
I20260927 21:07:11.047698 1661 raft_consensus.cc:2454] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e14d8196994c758d661b7a533c9024 in term 0.
I20260927 21:07:11.047866 1873 leader_election.cc:302] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e14d8196994c758d661b7a533c9024, 59ac76ad17a7412eb751079efe879b43; no voters:
I20260927 21:07:11.047999 2017 raft_consensus.cc:2781] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:11.048040 2017 raft_consensus.cc:482] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:11.048071 2017 raft_consensus.cc:3037] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:11.048530 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.044493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.98.3:33755)
0927 21:07:11.044617 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:11.044625 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:11.044661 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:11.045411 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:11.045412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:11.045414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:11.045497 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.045498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.046366 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.046367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.047586 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.047589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.048295 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.048296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.048362 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.048365 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:11.048426 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:07:11.048460 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260927 21:07:11.048547 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.044657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33755 (local address 127.0.98.2:36109)
0927 21:07:11.044699 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.044701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.044709 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:11.044737 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.044738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.044746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.044812 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:11.045586 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.046297 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.047678 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.048206 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.048293 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.048302 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.048449 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.048486 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:11.048887 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cb963477c540509e930389db88ebf9" candidate_uuid: "54e14d8196994c758d661b7a533c9024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2bf21e456b44c198a5003c48d464a4f" is_pre_election: true
I20260927 21:07:11.049021 1801 raft_consensus.cc:2454] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e14d8196994c758d661b7a533c9024 in term 0.
I20260927 21:07:11.049021 2017 raft_consensus.cc:504] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.049130 2017 leader_election.cc:288] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [CANDIDATE]: Term 1 election: Requested vote from peers 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627), c2bf21e456b44c198a5003c48d464a4f (127.0.98.2:36109)
I20260927 21:07:11.049278 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cb963477c540509e930389db88ebf9" candidate_uuid: "54e14d8196994c758d661b7a533c9024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2bf21e456b44c198a5003c48d464a4f"
I20260927 21:07:11.049360 1801 raft_consensus.cc:3037] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:11.049376 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94cb963477c540509e930389db88ebf9" candidate_uuid: "54e14d8196994c758d661b7a533c9024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ac76ad17a7412eb751079efe879b43"
I20260927 21:07:11.049456 1661 raft_consensus.cc:3037] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:11.050025 1661 raft_consensus.cc:2454] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e14d8196994c758d661b7a533c9024 in term 1.
I20260927 21:07:11.050125 1801 raft_consensus.cc:2454] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e14d8196994c758d661b7a533c9024 in term 1.
I20260927 21:07:11.050192 1873 leader_election.cc:302] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e14d8196994c758d661b7a533c9024, 59ac76ad17a7412eb751079efe879b43; no voters:
I20260927 21:07:11.050294 2017 raft_consensus.cc:2781] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:11.050396 2017 raft_consensus.cc:686] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 1 LEADER]: Becoming Leader. State: Replica: 54e14d8196994c758d661b7a533c9024, State: Running, Role: LEADER
I20260927 21:07:11.050483 2017 consensus_queue.cc:237] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.051218 1503 catalog_manager.cc:5495] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54e14d8196994c758d661b7a533c9024 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "54e14d8196994c758d661b7a533c9024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } health_report { overall_health: HEALTHY } } }
W20260927 21:07:11.067790 1708 tablet.cc:2369] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:07:11 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:11.083477 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.075687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33351 (local address 127.0.0.1:52904)
0927 21:07:11.075863 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:11.075874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:11.075954 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:11.076088 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:11.076089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:11.076492 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:11.076548 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.076551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.077230 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.077231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.078104 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.078106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.078435 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.078436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.078499 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.079334 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:11.079340 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:11.080786 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:11.082493 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:11.082499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:11.082500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:11.082680 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:11.082888 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:11.082888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:11.082889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:11.082940 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:11.083231 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:11.083237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:11.083277 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:07:11.083327 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:07:11.083407 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260927 21:07:11.083588 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.075774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.0.98.62:33351)
0927 21:07:11.075820 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.075823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.075954 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:07:11.076005 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.076005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.076013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.076062 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:11.076655 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.077152 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.078180 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.078356 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.078406 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.080875 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:11.080877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:11.080877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:11.080884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:11.082415 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:11.082755 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:11.082756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:11.082757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:11.082820 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:11.083002 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:11.083003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:11.083003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:11.083161 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:11.083224 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.083379 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.083466 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":25}
I20260927 21:07:11.085601 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52904:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:07:11.085829 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:11.087349 1781 tablet_service.cc:1467] Processing CreateTablet for tablet cda5453535af44fdb8d6be5cf6026408 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a81507a49214b4ea82657559138f8ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:11.087364 1641 tablet_service.cc:1467] Processing CreateTablet for tablet cda5453535af44fdb8d6be5cf6026408 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a81507a49214b4ea82657559138f8ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:11.087522 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda5453535af44fdb8d6be5cf6026408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:11.087524 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda5453535af44fdb8d6be5cf6026408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:11.088799 2012 tablet_bootstrap.cc:492] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: Bootstrap starting.
I20260927 21:07:11.089051 2011 tablet_bootstrap.cc:492] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: Bootstrap starting.
I20260927 21:07:11.089133 1921 tablet_service.cc:1467] Processing CreateTablet for tablet cda5453535af44fdb8d6be5cf6026408 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a81507a49214b4ea82657559138f8ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:11.089241 2012 tablet_bootstrap.cc:654] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:11.089262 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda5453535af44fdb8d6be5cf6026408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:11.089437 2011 tablet_bootstrap.cc:654] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:11.089973 2011 tablet_bootstrap.cc:492] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: No bootstrap required, opened a new log
I20260927 21:07:11.089973 2012 tablet_bootstrap.cc:492] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: No bootstrap required, opened a new log
I20260927 21:07:11.090018 2011 ts_tablet_manager.cc:1397] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:11.090018 2012 ts_tablet_manager.cc:1397] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:11.090155 2012 raft_consensus.cc:348] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.090155 2011 raft_consensus.cc:348] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.090215 2011 raft_consensus.cc:374] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:11.090215 2012 raft_consensus.cc:374] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:11.090236 2011 raft_consensus.cc:729] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ac76ad17a7412eb751079efe879b43, State: Initialized, Role: FOLLOWER
I20260927 21:07:11.090251 2012 raft_consensus.cc:729] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bf21e456b44c198a5003c48d464a4f, State: Initialized, Role: FOLLOWER
I20260927 21:07:11.090282 2011 consensus_queue.cc:260] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.090292 2012 consensus_queue.cc:260] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.090368 2012 ts_tablet_manager.cc:1428] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:11.090368 2011 ts_tablet_manager.cc:1428] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:11.090477 2028 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.090541 2028 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:11.090556 2018 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.090620 2018 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:11.090747 2010 tablet_bootstrap.cc:492] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: Bootstrap starting.
I20260927 21:07:11.091161 2010 tablet_bootstrap.cc:654] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:11.091826 2010 tablet_bootstrap.cc:492] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: No bootstrap required, opened a new log
I20260927 21:07:11.091884 2010 ts_tablet_manager.cc:1397] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:11.092018 2010 raft_consensus.cc:348] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.092083 2010 raft_consensus.cc:374] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:11.092115 2010 raft_consensus.cc:729] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e14d8196994c758d661b7a533c9024, State: Initialized, Role: FOLLOWER
I20260927 21:07:11.092170 2010 consensus_queue.cc:260] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.092278 2010 ts_tablet_manager.cc:1428] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:11.092424 2017 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.092492 2017 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 21:07:11.148156 1988 tablet.cc:2369] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:11.172879 2028 raft_consensus.cc:482] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:11.172983 2028 raft_consensus.cc:504] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.173138 2028 leader_election.cc:288] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627), 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231)
I20260927 21:07:11.173370 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda5453535af44fdb8d6be5cf6026408" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ac76ad17a7412eb751079efe879b43" is_pre_election: true
I20260927 21:07:11.173380 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda5453535af44fdb8d6be5cf6026408" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e14d8196994c758d661b7a533c9024" is_pre_election: true
I20260927 21:07:11.173477 1941 raft_consensus.cc:2454] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 0.
I20260927 21:07:11.173483 1661 raft_consensus.cc:2454] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 0.
I20260927 21:07:11.173645 1732 leader_election.cc:302] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e14d8196994c758d661b7a533c9024, c2bf21e456b44c198a5003c48d464a4f; no voters:
I20260927 21:07:11.173750 2028 raft_consensus.cc:2781] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:11.173805 2028 raft_consensus.cc:482] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:11.173821 2028 raft_consensus.cc:3037] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:11.174499 2028 raft_consensus.cc:504] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.174636 2028 leader_election.cc:288] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 election: Requested vote from peers 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627), 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231)
I20260927 21:07:11.174759 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda5453535af44fdb8d6be5cf6026408" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ac76ad17a7412eb751079efe879b43"
I20260927 21:07:11.174772 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda5453535af44fdb8d6be5cf6026408" candidate_uuid: "c2bf21e456b44c198a5003c48d464a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e14d8196994c758d661b7a533c9024"
I20260927 21:07:11.174851 1661 raft_consensus.cc:3037] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:11.174849 1941 raft_consensus.cc:3037] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:11.175496 1661 raft_consensus.cc:2454] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 1.
I20260927 21:07:11.175565 1941 raft_consensus.cc:2454] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2bf21e456b44c198a5003c48d464a4f in term 1.
I20260927 21:07:11.175698 1733 leader_election.cc:302] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ac76ad17a7412eb751079efe879b43, c2bf21e456b44c198a5003c48d464a4f; no voters:
I20260927 21:07:11.175817 2028 raft_consensus.cc:2781] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:11.175921 2028 raft_consensus.cc:686] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [term 1 LEADER]: Becoming Leader. State: Replica: c2bf21e456b44c198a5003c48d464a4f, State: Running, Role: LEADER
I20260927 21:07:11.175989 2028 consensus_queue.cc:237] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } }
I20260927 21:07:11.176293 2020 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: TxnStatusTablet state changed. Reason: New leader c2bf21e456b44c198a5003c48d464a4f. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.176404 2020 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:11.176613 1503 catalog_manager.cc:5495] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f reported cstate change: term changed from 0 to 1, leader changed from <none> to c2bf21e456b44c198a5003c48d464a4f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:11.176909 2052 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:07:11.177019 2052 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:07:11.186408 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.184052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.98.2:36109)
0927 21:07:11.184097 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.184099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.184108 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:11.184187 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.184188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.184199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.184272 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:11.184377 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.184859 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.185665 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.185862 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.185943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.186109 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.186149 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.186203 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.186308 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.186328 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:07:11.186409 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.183989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.0.1:41216)
0927 21:07:11.184057 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:11.184069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:11.184117 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:11.184278 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:11.184279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:11.184280 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:11.184325 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.184328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.184935 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.184936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.185572 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.185574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.185944 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.185945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.186006 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.186019 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.186218 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.186226 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:07:11.186291 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:07:11.186334 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:07:11.187314 1661 raft_consensus.cc:1261] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Refusing update from remote peer c2bf21e456b44c198a5003c48d464a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:11.187338 1941 raft_consensus.cc:1261] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Refusing update from remote peer c2bf21e456b44c198a5003c48d464a4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:11.187526 2020 consensus_queue.cc:1035] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:11.187641 2028 consensus_queue.cc:1035] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:11.188179 2017 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: TxnStatusTablet state changed. Reason: New leader c2bf21e456b44c198a5003c48d464a4f. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.188170 2018 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: TxnStatusTablet state changed. Reason: New leader c2bf21e456b44c198a5003c48d464a4f. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.188253 2017 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:11.188264 2018 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:11.188951 2028 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.189030 2028 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:11.189453 2017 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.189530 2017 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P 54e14d8196994c758d661b7a533c9024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:11.191619 2020 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.191715 2020 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P c2bf21e456b44c198a5003c48d464a4f: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:11.192072 2018 tablet_replica.cc:439] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2bf21e456b44c198a5003c48d464a4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 } } peers { permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 } } peers { permanent_uuid: "54e14d8196994c758d661b7a533c9024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42231 } } }
I20260927 21:07:11.192154 2018 tablet_replica.cc:442] T cda5453535af44fdb8d6be5cf6026408 P 59ac76ad17a7412eb751079efe879b43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:11.192291 2025 mvcc.cc:205] Tried to move back new op lower bound from 7334065074941648896 to 7334065074897133568. Current Snapshot: MvccSnapshot[applied={T|T < 7334065074941648896 or (T in {7334065074941648896})}]
I20260927 21:07:11.193130 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.190379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.0.98.3:42231)
0927 21:07:11.190439 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.190441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.190451 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:11.190493 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.190494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.190512 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.190592 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:11.190694 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.191241 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.192284 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.192568 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.192635 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.192786 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.192892 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.192977 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.193019 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.193038 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:07:11.197777 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.195481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53287 (local address 127.0.98.2:36109)
0927 21:07:11.195516 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.195518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.195526 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:11.195589 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.195590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.195598 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.195637 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:11.195755 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.196165 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.197203 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.197374 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.197421 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.197610 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.197632 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.197665 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.197717 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.197732 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:11.197834 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.195422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36109 (local address 127.0.98.3:53287)
0927 21:07:11.195492 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:11.195502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:11.195535 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:11.195649 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:11.195650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:11.195651 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:11.195703 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.195704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.196230 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.196231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.197124 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.197126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.197442 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.197443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.197513 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.197529 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.197665 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.197667 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:11.197716 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:07:11.197754 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:07:11.199949 1661 raft_consensus.cc:1261] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Refusing update from remote peer 54e14d8196994c758d661b7a533c9024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:11.200024 1801 raft_consensus.cc:1261] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Refusing update from remote peer 54e14d8196994c758d661b7a533c9024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:11.200184 2040 consensus_queue.cc:1035] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ac76ad17a7412eb751079efe879b43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:11.200279 2017 consensus_queue.cc:1035] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2bf21e456b44c198a5003c48d464a4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:11.208472 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.205950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42231 (local address 127.0.98.2:34955)
0927 21:07:11.206059 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:11.206070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:11.206114 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:11.206215 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:11.206215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:11.206216 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:11.206257 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.206258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.206801 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.206802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.207683 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:11.207685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.208137 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.208138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:11.208184 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.208195 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.208356 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.208357 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:11.208391 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 21:07:11.208425 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:07:11.208480 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.206039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34955 (local address 127.0.98.3:42231)
0927 21:07:11.206072 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.206074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.206101 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:07:11.206162 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.206163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.206171 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.206211 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:11.206309 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.206729 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.207791 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.208060 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.208122 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.208267 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:11.208299 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:11.208351 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.208394 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.208413 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:11.217372 1502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:11.217464 1502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:11.218041 1502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66906d99ca834483b1f5055ad8e4770d: Sending DeleteTablet for 3 replicas of tablet 94cb963477c540509e930389db88ebf9
I20260927 21:07:11.218304 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 94cb963477c540509e930389db88ebf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39538
I20260927 21:07:11.218313 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 94cb963477c540509e930389db88ebf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40072
I20260927 21:07:11.218348 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 94cb963477c540509e930389db88ebf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41198
I20260927 21:07:11.218425 2034 tablet_replica.cc:330] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: stopping tablet replica
I20260927 21:07:11.218418 2033 tablet_replica.cc:330] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: stopping tablet replica
I20260927 21:07:11.218482 2034 raft_consensus.cc:2229] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:11.218484 2033 raft_consensus.cc:2229] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:11.218519 2034 raft_consensus.cc:2258] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:11.218575 2033 raft_consensus.cc:2258] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:11.218667 2035 tablet_replica.cc:330] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: stopping tablet replica
I20260927 21:07:11.218725 2035 raft_consensus.cc:2229] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:11.218765 2035 raft_consensus.cc:2258] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:11.219134 2033 ts_tablet_manager.cc:1905] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:11.219205 2034 ts_tablet_manager.cc:1905] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:11.220978 2033 ts_tablet_manager.cc:1918] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:11.221030 2033 log.cc:1196] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/94cb963477c540509e930389db88ebf9
I20260927 21:07:11.221242 2033 ts_tablet_manager.cc:1939] T 94cb963477c540509e930389db88ebf9 P 54e14d8196994c758d661b7a533c9024: Deleting consensus metadata
I20260927 21:07:11.221274 2034 ts_tablet_manager.cc:1918] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:11.221326 2034 log.cc:1196] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/94cb963477c540509e930389db88ebf9
I20260927 21:07:11.221534 2034 ts_tablet_manager.cc:1939] T 94cb963477c540509e930389db88ebf9 P c2bf21e456b44c198a5003c48d464a4f: Deleting consensus metadata
I20260927 21:07:11.221535 1488 catalog_manager.cc:4841] TS 54e14d8196994c758d661b7a533c9024 (127.0.98.3:42231): tablet 94cb963477c540509e930389db88ebf9 (table test-table [id=e0df9927df644b7690a8cda12d6520ee]) successfully deleted
I20260927 21:07:11.221868 1489 catalog_manager.cc:4841] TS c2bf21e456b44c198a5003c48d464a4f (127.0.98.2:36109): tablet 94cb963477c540509e930389db88ebf9 (table test-table [id=e0df9927df644b7690a8cda12d6520ee]) successfully deleted
I20260927 21:07:11.221903 2035 ts_tablet_manager.cc:1905] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:11.223510 2035 ts_tablet_manager.cc:1918] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:11.223591 2035 log.cc:1196] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/94cb963477c540509e930389db88ebf9
I20260927 21:07:11.223798 2035 ts_tablet_manager.cc:1939] T 94cb963477c540509e930389db88ebf9 P 59ac76ad17a7412eb751079efe879b43: Deleting consensus metadata
I20260927 21:07:11.224110 1489 catalog_manager.cc:4841] TS 59ac76ad17a7412eb751079efe879b43 (127.0.98.1:41627): tablet 94cb963477c540509e930389db88ebf9 (table test-table [id=e0df9927df644b7690a8cda12d6520ee]) successfully deleted
Sep 27 21:07:11 dist-test-slave-8bgs krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:11.228919 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.219251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.1:41627)
0927 21:07:11.219292 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.219294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.219538 (+ 244us) server_negotiation.cc:407] Connection header received
0927 21:07:11.219630 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.219631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.219641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.219669 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:11.220485 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.221230 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.222376 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.222582 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.222666 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.225641 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:11.225651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:11.225651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:11.225658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:11.227497 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:11.227929 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:11.227933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:11.227933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:11.228012 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:11.228225 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:11.228226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:11.228226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:11.228416 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:11.228538 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.228787 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.228853 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:07:11.229434 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40074
I20260927 21:07:11.236509 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:11.230314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.0.98.62:33351)
0927 21:07:11.230372 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:11.230374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:11.230387 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:07:11.230481 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:11.230483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:11.230493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:11.230534 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:11.231018 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.231628 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.232427 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:11.232640 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:11.232720 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:11.233884 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:11.233885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:11.233886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:11.233893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:11.235340 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:11.235681 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:11.235682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:11.235683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:11.235741 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:11.235931 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:11.235932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:11.235932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:11.236042 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:11.236144 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:11.236246 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:07:11.236320 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 21:07:11.236693 1502 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52920
I20260927 21:07:11.237274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1576
I20260927 21:07:11.243240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1716
I20260927 21:07:11.249751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1856
I20260927 21:07:11.255575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 1472
2026-09-27T21:07:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6390 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:11 dist-test-slave-8bgs krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:11 dist-test-slave-8bgs krb5kdc[2070](info): set up 2 sockets
Sep 27 21:07:11 dist-test-slave-8bgs krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260927 21:07:12.286779 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:12.397650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:12.477706 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:12.477896 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:12.477918 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:12.479074 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:12.479169 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:07:12.479224 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:12.479244 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:12.479269 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:12.479291 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:12.480798 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45043
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:12.481040 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:12.481279 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:12.486071 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.486213 2083 server_base.cc:1034] running on GCE node
W20260927 21:07:12.486135 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:12.486163 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.486548 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:12.486835 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:12.487993 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790543232487969 us; error 35 us; skew 500 ppm
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:12.490396 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:12.490863 2083 webserver.cc:460] Webserver started at http://127.0.98.62:46495/ using document root <none> and password file <none>
I20260927 21:07:12.491145 2083 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:12.491214 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:12.491355 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:12.492436 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "62a33c7912974fd0900c72984e0d924c"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "295103ac8ef99b2dafc8b837c951b045"
server_key_iv: "5cf9626894ce6f301946c17bbbdfe7b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.492800 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "62a33c7912974fd0900c72984e0d924c"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "295103ac8ef99b2dafc8b837c951b045"
server_key_iv: "5cf9626894ce6f301946c17bbbdfe7b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.494074 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:12.494719 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.494882 2083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:12.494990 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "62a33c7912974fd0900c72984e0d924c"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "295103ac8ef99b2dafc8b837c951b045"
server_key_iv: "5cf9626894ce6f301946c17bbbdfe7b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.495069 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:12.511680 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:12.513973 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:12.514133 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:12.518885 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45043
I20260927 21:07:12.518941 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45043 every 8 connection(s)
I20260927 21:07:12.519310 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:12.519920 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:12.521971 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Bootstrap starting.
I20260927 21:07:12.522122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 2083
I20260927 21:07:12.522231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:12.522395 392 external_mini_cluster.cc:1428] Setting key 037b2986a4d3b10785e2921de37b9a6f
I20260927 21:07:12.522617 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:12.522957 2161 log.cc:824] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:12.523660 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: No bootstrap required, opened a new log
I20260927 21:07:12.524956 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a33c7912974fd0900c72984e0d924c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45043 } }
I20260927 21:07:12.525115 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:12.525161 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a33c7912974fd0900c72984e0d924c, State: Initialized, Role: FOLLOWER
I20260927 21:07:12.525278 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a33c7912974fd0900c72984e0d924c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45043 } }
I20260927 21:07:12.525367 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:12.525409 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:12.525454 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:12.525923 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a33c7912974fd0900c72984e0d924c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45043 } }
I20260927 21:07:12.526029 2161 leader_election.cc:302] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62a33c7912974fd0900c72984e0d924c; no voters:
I20260927 21:07:12.526180 2161 leader_election.cc:288] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:12.526203 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:12.526309 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [term 1 LEADER]: Becoming Leader. State: Replica: 62a33c7912974fd0900c72984e0d924c, State: Running, Role: LEADER
I20260927 21:07:12.526410 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:12.526444 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a33c7912974fd0900c72984e0d924c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45043 } }
I20260927 21:07:12.526818 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62a33c7912974fd0900c72984e0d924c. Latest consensus state: current_term: 1 leader_uuid: "62a33c7912974fd0900c72984e0d924c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a33c7912974fd0900c72984e0d924c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45043 } } }
I20260927 21:07:12.527107 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:12.527011 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62a33c7912974fd0900c72984e0d924c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a33c7912974fd0900c72984e0d924c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45043 } } }
I20260927 21:07:12.527556 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:12.527704 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:12.528232 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:12.529698 2175 catalog_manager.cc:1347] Generated new cluster ID: a22550aa6a454c7282f902d2bf3d511d
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:12.529747 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:12.533670 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.522890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.98.62:45043)
0927 21:07:12.523068 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.523077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.523126 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:07:12.523574 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.523580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.523625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.523768 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.524372 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.525697 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.526637 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.527291 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.527414 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.530296 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.530299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.530299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.530310 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.532112 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.532498 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.532501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.532501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.532596 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.532827 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.532828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.532828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.533086 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.533214 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.533370 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.533485 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260927 21:07:12.536516 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:12.536986 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:12.542851 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Generated new TSK 0
I20260927 21:07:12.543007 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:12.571888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45043
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:12.650990 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:12.651262 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:12.651309 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:12.652449 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:12.652513 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:12.652549 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:07:12.652582 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:12.652618 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:12.654059 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:12.654322 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:12.654543 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:12.659441 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:12.659426 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:12.659427 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.659452 2187 server_base.cc:1034] running on GCE node
I20260927 21:07:12.659801 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:12.659984 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:12.661118 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790543232661107 us; error 27 us; skew 500 ppm
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:12.663453 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:12.663889 2187 webserver.cc:460] Webserver started at http://127.0.98.1:42389/ using document root <none> and password file <none>
I20260927 21:07:12.664115 2187 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:12.664160 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:12.664299 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:12.665306 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed0191a61f194b03976d39be8c89ef21"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "e84f39a1847c087cd91aa5c1694006ec"
server_key_iv: "dde8422a9e93e41233de18160ddf16e2"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.665660 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed0191a61f194b03976d39be8c89ef21"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "e84f39a1847c087cd91aa5c1694006ec"
server_key_iv: "dde8422a9e93e41233de18160ddf16e2"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.666939 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:12.667676 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.667887 2187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.667971 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "ed0191a61f194b03976d39be8c89ef21"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "e84f39a1847c087cd91aa5c1694006ec"
server_key_iv: "dde8422a9e93e41233de18160ddf16e2"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.668040 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:12.676964 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:12.678580 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:12.678730 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:12.679255 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:12.679294 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.679314 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:12.679327 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.685743 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34037
I20260927 21:07:12.685824 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34037 every 8 connection(s)
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:12.686125 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:12.689112 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.679590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35327 (local address 127.0.98.62:45043)
0927 21:07:12.679651 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.679653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.679914 (+ 261us) server_negotiation.cc:407] Connection header received
0927 21:07:12.680332 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.680334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.680348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.680427 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.681155 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.682027 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.683505 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.683901 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.683988 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.686579 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.686581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.686581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.686588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.688012 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.688366 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.688366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.688367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.688414 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.688621 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.688622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.688622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.688715 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.688776 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.688992 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.689051 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:07:12.689249 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.679577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.98.1:35327)
0927 21:07:12.679785 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:12.679823 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:12.680248 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:12.680481 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:12.680484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:12.680793 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:12.681054 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.681059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.682123 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.682124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.683403 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.683405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.684008 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.684009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.684157 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.684647 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:12.684656 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:12.686475 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:12.688081 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.688085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.688087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.688298 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.688469 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.688470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.688470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.688568 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.688785 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:12.688788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:12.688890 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 21:07:12.688969 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:07:12.689079 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":149,"threads_started":1}
I20260927 21:07:12.689708 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45043
I20260927 21:07:12.689787 2318 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:12.689954 2318 heartbeater.cc:511] Master 127.0.98.62:45043 requested a full tablet report, sending...
I20260927 21:07:12.690524 2115 ts_manager.cc:194] Registered new tserver with Master: ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037)
I20260927 21:07:12.691502 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35327
I20260927 21:07:12.695636 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.690736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52887 (local address 127.0.98.62:45043)
0927 21:07:12.690794 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.690796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.690898 (+ 102us) server_negotiation.cc:407] Connection header received
0927 21:07:12.690974 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.690974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.690982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.691022 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.691379 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.691991 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.692669 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.692860 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.692916 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.693722 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.693723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.693723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.693729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.694795 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.695077 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.695078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.695078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.695162 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.695313 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.695313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.695313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.695391 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.695433 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.695511 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.695581 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:12.695644 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.690700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.98.1:52887)
0927 21:07:12.690848 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:12.690858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:12.690911 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:12.691033 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:12.691033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:12.691247 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:12.691317 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.691318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.692087 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.692087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.692591 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.692592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.692925 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.692926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.692977 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.693384 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:12.693389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:12.693650 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:12.694861 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.694862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.694862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.695020 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.695225 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.695225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.695226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.695273 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.695424 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:12.695425 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:12.695454 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:07:12.695496 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 21:07:12.695578 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260927 21:07:12.696293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 2187
I20260927 21:07:12.696395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:12.696563 392 external_mini_cluster.cc:1428] Setting key c265138bae562256f3308feb436a2cc6
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:12.717674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45043
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:12.796402 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:12.796586 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:12.796609 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:12.797741 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:12.797780 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:12.797799 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:07:12.797811 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:12.797864 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:12.799283 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:12.799505 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:12.799719 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:12.804641 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.804679 2327 server_base.cc:1034] running on GCE node
W20260927 21:07:12.804661 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:12.804647 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.805092 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:12.805341 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:12.806502 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790543232806496 us; error 35 us; skew 500 ppm
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:12.808930 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:12.809333 2327 webserver.cc:460] Webserver started at http://127.0.98.2:45531/ using document root <none> and password file <none>
I20260927 21:07:12.809549 2327 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:12.809614 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:12.809764 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:12.810739 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "dc03939d49ca43ffa216cf3d2450bb07"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "ea961bccefc167c06b3a60e9a2dc2b84"
server_key_iv: "6086b325c6f13f11e22bf45eecf6d555"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.811168 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dc03939d49ca43ffa216cf3d2450bb07"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "ea961bccefc167c06b3a60e9a2dc2b84"
server_key_iv: "6086b325c6f13f11e22bf45eecf6d555"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.812463 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:12.813160 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.813333 2327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:12.813426 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "dc03939d49ca43ffa216cf3d2450bb07"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "ea961bccefc167c06b3a60e9a2dc2b84"
server_key_iv: "6086b325c6f13f11e22bf45eecf6d555"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.813508 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:12.823151 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:12.825352 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:12.825508 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:12.825987 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:12.826053 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.826108 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:12.826141 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.833223 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42871
I20260927 21:07:12.833292 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42871 every 8 connection(s)
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:12.833575 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:12.836593 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.826518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49617 (local address 127.0.98.62:45043)
0927 21:07:12.826580 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.826583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.826805 (+ 222us) server_negotiation.cc:407] Connection header received
0927 21:07:12.827415 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.827416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.827457 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.827523 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.828320 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.829159 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.830824 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.831168 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.831250 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.833816 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.833818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.833818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.833838 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.835241 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.835653 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.835655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.835656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.835724 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.835969 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.835969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.835970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.836085 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.836155 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.836432 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.836505 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:07:12.836781 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.826501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.98.2:49617)
0927 21:07:12.826671 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:12.826709 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:12.827289 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:12.827594 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:12.827598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:12.827948 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:12.828240 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.828247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.829279 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.829280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.830725 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.830728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.831270 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.831271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.831420 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.831960 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:12.831971 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:12.833706 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:12.835329 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.835335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.835337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.835562 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.835790 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.835791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.835791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.835901 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.836157 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:12.836163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:12.836297 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:07:12.836413 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:07:12.836555 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260927 21:07:12.837293 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45043
I20260927 21:07:12.837364 2458 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:12.837538 2458 heartbeater.cc:511] Master 127.0.98.62:45043 requested a full tablet report, sending...
I20260927 21:07:12.837988 2114 ts_manager.cc:194] Registered new tserver with Master: dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871)
I20260927 21:07:12.838364 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49617
I20260927 21:07:12.841727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 2327
I20260927 21:07:12.841825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:12.842001 392 external_mini_cluster.cc:1428] Setting key c0bc31e6c5eb4dea41104ac388f601ae
I20260927 21:07:12.843693 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.838413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45839 (local address 127.0.98.62:45043)
0927 21:07:12.838482 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.838484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.838583 (+ 99us) server_negotiation.cc:407] Connection header received
0927 21:07:12.838647 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.838648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.838658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.838717 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.839042 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.839537 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.840151 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.840319 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.840366 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.841073 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.841074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.841074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.841079 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.842476 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.842860 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.842862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.842862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.842915 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.843161 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.843162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.843162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.843298 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.843377 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.843535 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.843605 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:07:12.843708 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.838358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.98.2:45839)
0927 21:07:12.838525 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:12.838535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:12.838600 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:12.838724 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:12.838724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:12.838919 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:12.838972 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.838973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.839617 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.839618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.840087 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.840089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.840379 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.840380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.840434 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.840760 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:12.840764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:12.841004 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:12.842570 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.842571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.842572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.842774 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.842989 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.842989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.842990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.843045 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.843370 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:12.843370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:12.843413 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:07:12.843501 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:07:12.843607 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:12.863547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45043
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:12.942651 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:12.942843 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:12.942866 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:12.944011 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:12.944051 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:12.944072 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:07:12.944083 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:12.944140 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:12.945535 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:12.945765 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:12.945986 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:12.951112 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.951351 2467 server_base.cc:1034] running on GCE node
W20260927 21:07:12.954753 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:12.954929 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:12.955186 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:12.955466 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:12.956673 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790543232956629 us; error 34 us; skew 500 ppm
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:12.960009 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:12.960487 2467 webserver.cc:460] Webserver started at http://127.0.98.3:34807/ using document root <none> and password file <none>
I20260927 21:07:12.960731 2467 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:12.960798 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:12.960932 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:12.962234 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "42d338e474ae46188de9b34d3eaf97b0"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "36726d505842324a6134d2d40e1e0a14"
server_key_iv: "f7c1b58f31812ae1052596bcb24fd28e"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.962585 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42d338e474ae46188de9b34d3eaf97b0"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "36726d505842324a6134d2d40e1e0a14"
server_key_iv: "f7c1b58f31812ae1052596bcb24fd28e"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.963967 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:12.964646 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.964828 2467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:12.964919 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "42d338e474ae46188de9b34d3eaf97b0"
format_stamp: "Formatted at 2026-09-27 21:07:12 on dist-test-slave-8bgs"
server_key: "36726d505842324a6134d2d40e1e0a14"
server_key_iv: "f7c1b58f31812ae1052596bcb24fd28e"
server_key_version: "encryptionkey@0"
I20260927 21:07:12.965008 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:12.979487 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:12.981581 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:12.981730 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:12.982221 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:12.982276 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:12.982347 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:12.982379 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:07:12 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:12.989444 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44999
I20260927 21:07:12.989521 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44999 every 8 connection(s)
I20260927 21:07:12.989792 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:12.992040 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.982586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44935 (local address 127.0.98.62:45043)
0927 21:07:12.982633 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.982635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.982904 (+ 269us) server_negotiation.cc:407] Connection header received
0927 21:07:12.983576 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.983578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.983588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.983631 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.984473 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.984953 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.986595 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.986802 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.986887 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.989281 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.989282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.989282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.989290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.990922 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.991332 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.991332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.991333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.991397 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.991608 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.991609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.991609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.991699 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.991739 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.991922 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.991978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:12.992182 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.982585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.98.3:44935)
0927 21:07:12.982756 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:12.982802 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:12.983468 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:12.983705 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:12.983710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:12.984086 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:12.984363 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.984370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.985019 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.985020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.986501 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.986504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.986902 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.986903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.987016 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.987704 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:12.987714 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:12.989175 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:12.990991 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.990994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.990996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.991268 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.991454 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.991454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.991455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.991561 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.991741 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:12.991743 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:12.991840 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 21:07:12.991903 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:07:12.992008 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260927 21:07:12.992651 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45043
I20260927 21:07:12.992719 2598 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:12.992901 2598 heartbeater.cc:511] Master 127.0.98.62:45043 requested a full tablet report, sending...
I20260927 21:07:12.993218 2115 ts_manager.cc:194] Registered new tserver with Master: 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:12.993604 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44935
I20260927 21:07:12.997802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 2467
I20260927 21:07:12.997906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:12.998095 392 external_mini_cluster.cc:1428] Setting key 1c58477a726818604b1ef8fe2434203e
I20260927 21:07:12.999063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:07:12.999555 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.993688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39949 (local address 127.0.98.62:45043)
0927 21:07:12.993726 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:12.993727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:12.993885 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:07:12.993954 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:12.993955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:12.993965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:12.994027 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:12.994332 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.994794 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.995557 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.995746 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.995827 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.996684 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:12.996686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:12.996686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:12.996692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:12.998326 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.998744 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.998745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.998745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.998823 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:12.999019 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:12.999020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:12.999020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:12.999177 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:12.999242 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:12.999396 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:07:12.999466 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:12.999569 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:12.993639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.98.3:39949)
0927 21:07:12.993815 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:12.993825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:12.993898 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:12.994043 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:12.994044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:12.994209 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:12.994257 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.994258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.994883 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.994885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.995462 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:12.995464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:12.995829 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:12.995830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:12.995908 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:12.996308 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:12.996313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:12.996578 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:12.998443 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.998444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.998445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.998672 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.998888 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:12.998889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:12.998889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:12.998954 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:12.999239 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:12.999240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:12.999283 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:07:12.999370 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:07:12.999467 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":134,"threads_started":1}
Sep 27 21:07:13 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:07:13 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:13.020704 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.014172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.0.98.62:45043)
0927 21:07:13.014230 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.014232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.014242 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:13.014270 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.014271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.014281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.014346 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:13.014874 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.015408 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.016177 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.016368 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.016422 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.018448 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:13.018449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:13.018450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:13.018455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:13.019892 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:13.020173 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:13.020174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:13.020174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:13.020222 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:13.020363 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:13.020363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:13.020364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:13.020449 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:13.020518 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.020591 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.020643 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:07:13.022157 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:13.023842 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:13.030092 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.025849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.0.1:36944)
0927 21:07:13.026076 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.026087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.026136 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.026526 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.026527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.026529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.026589 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.026591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.027881 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.027882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.028996 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.028998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.029843 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.029844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.029899 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.029909 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.029960 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.029997 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":73,"thread_start_us":55,"threads_started":1}
I20260927 21:07:13.030122 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.025991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.0.98.3:44999)
0927 21:07:13.026372 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.026376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.026384 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:13.026417 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.026419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.026432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.026665 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.026716 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.027667 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.029075 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.029763 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.029840 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.029896 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.029985 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.030031 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":92,"threads_started":1}
I20260927 21:07:13.030501 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.026101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.98.1:34037)
0927 21:07:13.026306 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.026311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.026321 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:13.026360 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.026362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.026379 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.026463 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.026692 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.027876 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.029285 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.029979 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.030043 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.030098 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.030349 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.030391 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260927 21:07:13.030289 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.025869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:33274)
0927 21:07:13.026205 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.026214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.026250 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.026515 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.026518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.026520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.026585 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.026586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.027971 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.027972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.029199 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.029202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.030058 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.030058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.030112 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.030116 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.030169 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.030217 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":65,"thread_start_us":51,"threads_started":1}
I20260927 21:07:13.031133 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.025849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42871 (local address 127.0.0.1:44502)
0927 21:07:13.025996 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.026052 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.026114 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.026593 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.026594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.026595 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.026629 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.026631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.027971 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.027972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.029200 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.029202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.030851 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.030852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.030927 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.030931 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.030987 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.031024 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":56,"thread_start_us":52,"threads_started":1}
I20260927 21:07:13.031272 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 5afcb477119e4d329a428b27f00c9169 (DEFAULT_TABLE table=test-table [id=9e1c992bab6042e983d0805b2d8a4cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:13.031433 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 5afcb477119e4d329a428b27f00c9169 (DEFAULT_TABLE table=test-table [id=9e1c992bab6042e983d0805b2d8a4cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:13.031669 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afcb477119e4d329a428b27f00c9169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.031792 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afcb477119e4d329a428b27f00c9169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.034240 2618 tablet_bootstrap.cc:492] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Bootstrap starting.
I20260927 21:07:13.034561 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.026186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.0.98.2:42871)
0927 21:07:13.026369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.026373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.026383 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:13.026419 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.026423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.026462 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.026575 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.026692 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.027868 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.030134 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.030753 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.030847 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.030905 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.030999 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.034464 (+ 3465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
I20260927 21:07:13.034767 2619 tablet_bootstrap.cc:492] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Bootstrap starting.
I20260927 21:07:13.035189 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 5afcb477119e4d329a428b27f00c9169 (DEFAULT_TABLE table=test-table [id=9e1c992bab6042e983d0805b2d8a4cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:13.035317 2618 tablet_bootstrap.cc:654] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.035382 2619 tablet_bootstrap.cc:654] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.035440 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afcb477119e4d329a428b27f00c9169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.035671 2618 log.cc:824] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:13.035679 2619 log.cc:824] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:13.036404 2618 tablet_bootstrap.cc:492] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: No bootstrap required, opened a new log
I20260927 21:07:13.036458 2618 ts_tablet_manager.cc:1397] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:13.036610 2619 tablet_bootstrap.cc:492] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: No bootstrap required, opened a new log
I20260927 21:07:13.036655 2619 ts_tablet_manager.cc:1397] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:07:13.038339 2618 raft_consensus.cc:348] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.038491 2618 raft_consensus.cc:374] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.038522 2618 raft_consensus.cc:729] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d338e474ae46188de9b34d3eaf97b0, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.038443 2619 raft_consensus.cc:348] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.038561 2619 raft_consensus.cc:374] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.038570 2622 tablet_bootstrap.cc:492] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Bootstrap starting.
I20260927 21:07:13.038589 2619 raft_consensus.cc:729] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.038638 2618 consensus_queue.cc:260] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.038710 2619 consensus_queue.cc:260] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.038889 2618 ts_tablet_manager.cc:1428] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:07:13.038947 2619 ts_tablet_manager.cc:1428] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 21:07:13.038995 2598 heartbeater.cc:503] Master 127.0.98.62:45043 was elected leader, sending a full tablet report...
I20260927 21:07:13.039036 2318 heartbeater.cc:503] Master 127.0.98.62:45043 was elected leader, sending a full tablet report...
I20260927 21:07:13.039214 2622 tablet_bootstrap.cc:654] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.039539 2622 log.cc:824] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:13.040344 2622 tablet_bootstrap.cc:492] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: No bootstrap required, opened a new log
I20260927 21:07:13.040396 2622 ts_tablet_manager.cc:1397] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:07:13.041611 2622 raft_consensus.cc:348] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.041730 2622 raft_consensus.cc:374] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.041759 2622 raft_consensus.cc:729] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc03939d49ca43ffa216cf3d2450bb07, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.041834 2622 consensus_queue.cc:260] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.042053 2622 ts_tablet_manager.cc:1428] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:07:13.042083 2458 heartbeater.cc:503] Master 127.0.98.62:45043 was elected leader, sending a full tablet report...
I20260927 21:07:13.057467 2624 raft_consensus.cc:482] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:13.057547 2624 raft_consensus.cc:504] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.057895 2624 leader_election.cc:288] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999), dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871)
I20260927 21:07:13.061584 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.058120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59171 (local address 127.0.98.3:44999)
0927 21:07:13.058176 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.058178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.058187 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:13.058211 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.058212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.058223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.058314 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.058476 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.059259 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.060741 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.061267 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.061332 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.061340 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.061487 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.061522 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:07:13.061584 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.058139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40007 (local address 127.0.98.2:42871)
0927 21:07:13.058187 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.058189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.058294 (+ 105us) server_negotiation.cc:407] Connection header received
0927 21:07:13.058352 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.058353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.058364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.058409 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.058557 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.059270 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.060742 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.061259 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.061332 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.061340 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.061487 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.061522 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:07:13.061605 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.058055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.98.1:59171)
0927 21:07:13.058093 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.058105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.058143 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.058317 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.058318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.058319 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.058389 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.058390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.059341 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.059342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.060657 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.060659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.061339 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.061340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.061410 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.061413 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.061467 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.061512 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":99}
I20260927 21:07:13.061617 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.058055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42871 (local address 127.0.98.1:40007)
0927 21:07:13.058224 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.058234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.058304 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.058428 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.058429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.058430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.058491 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.058492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.059341 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.059342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.060657 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.060659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.061339 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.061340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.061410 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.061413 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.061467 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.061512 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":92,"thread_start_us":82,"threads_started":1}
I20260927 21:07:13.061923 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afcb477119e4d329a428b27f00c9169" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0" is_pre_election: true
I20260927 21:07:13.062023 2552 raft_consensus.cc:2454] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 0.
I20260927 21:07:13.062224 2206 leader_election.cc:302] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:13.062196 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afcb477119e4d329a428b27f00c9169" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07" is_pre_election: true
I20260927 21:07:13.062335 2412 raft_consensus.cc:2454] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 0.
I20260927 21:07:13.062362 2624 raft_consensus.cc:2781] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:13.062498 2624 raft_consensus.cc:482] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:13.062529 2624 raft_consensus.cc:3037] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.063249 2624 raft_consensus.cc:504] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.063387 2624 leader_election.cc:288] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 election: Requested vote from peers 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999), dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871)
I20260927 21:07:13.063529 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afcb477119e4d329a428b27f00c9169" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0"
I20260927 21:07:13.063536 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afcb477119e4d329a428b27f00c9169" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07"
I20260927 21:07:13.063613 2412 raft_consensus.cc:3037] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.063617 2552 raft_consensus.cc:3037] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.064536 2552 raft_consensus.cc:2454] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 1.
I20260927 21:07:13.064625 2412 raft_consensus.cc:2454] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 1.
I20260927 21:07:13.064697 2206 leader_election.cc:302] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:13.064780 2624 raft_consensus.cc:2781] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:13.064898 2624 raft_consensus.cc:686] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 1 LEADER]: Becoming Leader. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Running, Role: LEADER
I20260927 21:07:13.064980 2624 consensus_queue.cc:237] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } }
I20260927 21:07:13.065650 2114 catalog_manager.cc:5495] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed0191a61f194b03976d39be8c89ef21 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: HEALTHY } } }
W20260927 21:07:13.084101 2459 tablet.cc:2369] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:07:13 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:13.109932 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.103785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45043 (local address 127.0.0.1:49276)
0927 21:07:13.103942 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.103951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.104011 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.104132 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.104133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.104408 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:13.104462 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.104464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.105058 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.105059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.105555 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.105556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.105851 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.105852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.105911 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.106414 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:13.106419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:13.107797 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:13.109060 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:13.109066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:13.109068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:13.109248 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:13.109410 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:13.109411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:13.109411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:13.109470 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:13.109660 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:13.109665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:13.109703 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.109759 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.109842 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260927 21:07:13.109995 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.103876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.98.62:45043)
0927 21:07:13.103941 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.103943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.104011 (+ 68us) server_negotiation.cc:407] Connection header received
0927 21:07:13.104064 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.104065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.104072 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.104112 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:13.104525 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.104996 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.105619 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.105792 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.105839 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.107867 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:13.107868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:13.107869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:13.107874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:13.108994 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:13.109310 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:13.109311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:13.109311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:13.109356 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:13.109525 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:13.109525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:13.109525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:13.109609 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:13.109655 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.109786 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.109883 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":45}
I20260927 21:07:13.111713 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49276:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:07:13.111927 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:13.113389 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 626a5903f8d6439faf2e4d20f78cc028 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bff60651cd3644c4b2588234212bce32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:13.113507 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 626a5903f8d6439faf2e4d20f78cc028 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bff60651cd3644c4b2588234212bce32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:13.113565 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626a5903f8d6439faf2e4d20f78cc028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.113653 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626a5903f8d6439faf2e4d20f78cc028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.115139 2618 tablet_bootstrap.cc:492] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: Bootstrap starting.
I20260927 21:07:13.115144 2619 tablet_bootstrap.cc:492] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: Bootstrap starting.
I20260927 21:07:13.115258 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 626a5903f8d6439faf2e4d20f78cc028 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bff60651cd3644c4b2588234212bce32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:13.115418 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626a5903f8d6439faf2e4d20f78cc028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.115618 2619 tablet_bootstrap.cc:654] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.115684 2618 tablet_bootstrap.cc:654] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.116338 2619 tablet_bootstrap.cc:492] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: No bootstrap required, opened a new log
I20260927 21:07:13.116384 2619 ts_tablet_manager.cc:1397] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:13.116384 2618 tablet_bootstrap.cc:492] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: No bootstrap required, opened a new log
I20260927 21:07:13.116443 2618 ts_tablet_manager.cc:1397] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:13.116523 2619 raft_consensus.cc:348] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.116600 2619 raft_consensus.cc:374] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.116614 2619 raft_consensus.cc:729] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.116582 2618 raft_consensus.cc:348] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.116643 2618 raft_consensus.cc:374] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.116662 2618 raft_consensus.cc:729] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d338e474ae46188de9b34d3eaf97b0, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.116668 2619 consensus_queue.cc:260] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.116705 2618 consensus_queue.cc:260] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.116740 2622 tablet_bootstrap.cc:492] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: Bootstrap starting.
I20260927 21:07:13.116755 2619 ts_tablet_manager.cc:1428] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:13.116799 2618 ts_tablet_manager.cc:1428] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:13.116870 2624 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.116943 2624 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.116910 2623 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.116988 2623 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.117465 2622 tablet_bootstrap.cc:654] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.118088 2622 tablet_bootstrap.cc:492] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: No bootstrap required, opened a new log
I20260927 21:07:13.118150 2622 ts_tablet_manager.cc:1397] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:13.118294 2622 raft_consensus.cc:348] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.118366 2622 raft_consensus.cc:374] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.118397 2622 raft_consensus.cc:729] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc03939d49ca43ffa216cf3d2450bb07, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.118450 2622 consensus_queue.cc:260] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.118549 2622 ts_tablet_manager.cc:1428] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:13.118616 2626 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.118742 2626 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.161752 2624 raft_consensus.cc:482] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:13.161852 2624 raft_consensus.cc:504] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.162039 2624 leader_election.cc:288] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871), 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:13.162240 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626a5903f8d6439faf2e4d20f78cc028" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07" is_pre_election: true
I20260927 21:07:13.162240 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626a5903f8d6439faf2e4d20f78cc028" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0" is_pre_election: true
I20260927 21:07:13.162345 2552 raft_consensus.cc:2454] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 0.
I20260927 21:07:13.162345 2412 raft_consensus.cc:2454] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 0.
I20260927 21:07:13.162482 2206 leader_election.cc:302] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:13.162590 2624 raft_consensus.cc:2781] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:13.162632 2624 raft_consensus.cc:482] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:13.162647 2624 raft_consensus.cc:3037] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.163375 2624 raft_consensus.cc:504] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.163499 2624 leader_election.cc:288] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 election: Requested vote from peers dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871), 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:13.163686 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626a5903f8d6439faf2e4d20f78cc028" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07"
I20260927 21:07:13.163705 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626a5903f8d6439faf2e4d20f78cc028" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0"
I20260927 21:07:13.163774 2552 raft_consensus.cc:3037] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.163774 2412 raft_consensus.cc:3037] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.164515 2552 raft_consensus.cc:2454] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 1.
I20260927 21:07:13.164515 2412 raft_consensus.cc:2454] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 1.
I20260927 21:07:13.164667 2206 leader_election.cc:302] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:13.164764 2624 raft_consensus.cc:2781] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:13.164814 2624 raft_consensus.cc:686] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [term 1 LEADER]: Becoming Leader. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Running, Role: LEADER
I20260927 21:07:13.164853 2624 consensus_queue.cc:237] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.165162 2628 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: TxnStatusTablet state changed. Reason: New leader ed0191a61f194b03976d39be8c89ef21. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.165238 2628 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:13.165460 2640 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:07:13.165438 2114 catalog_manager.cc:5495] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed0191a61f194b03976d39be8c89ef21 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:13.165611 2640 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:07:13.176712 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.173968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:33280)
0927 21:07:13.174069 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.174080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.174135 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.174297 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.174297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.174299 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:13.174341 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.174343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.175223 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.175223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.175768 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.175769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.176230 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.176231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.176280 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.176292 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.176586 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.176593 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.176622 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.176657 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:07:13.176716 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.174016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.98.1:34037)
0927 21:07:13.174088 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.174090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.174120 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:07:13.174192 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.174193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.174211 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.174299 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:13.174381 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.175144 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.175841 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.176150 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.176222 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.176376 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.176524 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.176603 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.176626 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.176646 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:07:13.178478 2412 raft_consensus.cc:1261] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Refusing update from remote peer ed0191a61f194b03976d39be8c89ef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:13.178498 2552 raft_consensus.cc:1261] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Refusing update from remote peer ed0191a61f194b03976d39be8c89ef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:13.178812 2624 consensus_queue.cc:1035] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:13.178943 2628 consensus_queue.cc:1035] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:13.180295 2623 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: TxnStatusTablet state changed. Reason: New leader ed0191a61f194b03976d39be8c89ef21. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.180403 2623 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.180478 2626 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: TxnStatusTablet state changed. Reason: New leader ed0191a61f194b03976d39be8c89ef21. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.180583 2626 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.181231 2624 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.181326 2624 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:13.181257 2648 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.181393 2643 mvcc.cc:205] Tried to move back new op lower bound from 7334065083096739840 to 7334065083043430400. Current Snapshot: MvccSnapshot[applied={T|T < 7334065083096739840}]
I20260927 21:07:13.181391 2648 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P ed0191a61f194b03976d39be8c89ef21: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:13.181630 2644 mvcc.cc:205] Tried to move back new op lower bound from 7334065083096739840 to 7334065083043430400. Current Snapshot: MvccSnapshot[applied={T|T < 7334065083096739840}]
I20260927 21:07:13.181706 2626 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.181784 2626 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P dc03939d49ca43ffa216cf3d2450bb07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.182077 2623 tablet_replica.cc:439] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } } }
I20260927 21:07:13.182165 2623 tablet_replica.cc:442] T 626a5903f8d6439faf2e4d20f78cc028 P 42d338e474ae46188de9b34d3eaf97b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:13.182371 2645 mvcc.cc:205] Tried to move back new op lower bound from 7334065083096739840 to 7334065083043430400. Current Snapshot: MvccSnapshot[applied={T|T < 7334065083096739840}]
I20260927 21:07:13.186532 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.184141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.1:34037)
0927 21:07:13.184188 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.184190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.184198 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:13.184224 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.184225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.184246 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.184311 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:13.184403 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.185104 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.185805 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.186100 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.186170 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.186289 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.186320 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.186381 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.186430 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.186457 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 21:07:13.186861 2319 tablet.cc:2369] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:13.191283 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.188697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.1:59291)
0927 21:07:13.188774 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.188784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.188834 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.188945 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.188946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.188947 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:13.188993 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.188994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.189838 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.189839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.190461 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.190463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.190890 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.190891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.190933 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.190944 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.191134 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.191140 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.191174 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.191205 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:07:13.191291 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.188759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59291 (local address 127.0.98.1:34037)
0927 21:07:13.188790 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.188792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.188824 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:07:13.188873 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.188873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.188881 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.188944 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:13.189027 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.189759 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.190537 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.190824 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.190923 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.191024 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.191051 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.191119 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.191185 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.191200 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:13.193679 2412 raft_consensus.cc:1261] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Refusing update from remote peer ed0191a61f194b03976d39be8c89ef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:13.193867 2628 consensus_queue.cc:1035] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:13.195354 2552 raft_consensus.cc:1261] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Refusing update from remote peer ed0191a61f194b03976d39be8c89ef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:13.195542 2628 consensus_queue.cc:1035] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:13.209080 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:13.209179 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:13.209896 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Sending DeleteTablet for 3 replicas of tablet 5afcb477119e4d329a428b27f00c9169
I20260927 21:07:13.210201 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afcb477119e4d329a428b27f00c9169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36944
I20260927 21:07:13.210204 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afcb477119e4d329a428b27f00c9169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 21:07:13.210489 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afcb477119e4d329a428b27f00c9169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44502
I20260927 21:07:13.210453 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:13.210582 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:13.210704 2661 tablet_replica.cc:330] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: stopping tablet replica
I20260927 21:07:13.210808 2661 raft_consensus.cc:2229] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:13.210863 2661 raft_consensus.cc:2258] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:13.211277 2661 ts_tablet_manager.cc:1905] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:13.211504 2660 tablet_replica.cc:330] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: stopping tablet replica
I20260927 21:07:13.211642 2659 tablet_replica.cc:330] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: stopping tablet replica
I20260927 21:07:13.211685 2532 tablet_service.cc:1467] Processing CreateTablet for tablet a59c938868d04bd6a4e92541391bf851 (DEFAULT_TABLE table=test-table [id=961a4e3b2b2147e596ad37a062548647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:13.211819 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c938868d04bd6a4e92541391bf851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.211961 2392 tablet_service.cc:1467] Processing CreateTablet for tablet a59c938868d04bd6a4e92541391bf851 (DEFAULT_TABLE table=test-table [id=961a4e3b2b2147e596ad37a062548647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:13.212042 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c938868d04bd6a4e92541391bf851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.213056 2661 ts_tablet_manager.cc:1918] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:13.213116 2661 log.cc:1196] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/5afcb477119e4d329a428b27f00c9169
I20260927 21:07:13.213243 2618 tablet_bootstrap.cc:492] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Bootstrap starting.
I20260927 21:07:13.213261 2622 tablet_bootstrap.cc:492] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Bootstrap starting.
I20260927 21:07:13.213614 2618 tablet_bootstrap.cc:654] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.213653 2622 tablet_bootstrap.cc:654] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.213917 2659 raft_consensus.cc:2229] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:13.214111 2661 ts_tablet_manager.cc:1939] T 5afcb477119e4d329a428b27f00c9169 P dc03939d49ca43ffa216cf3d2450bb07: Deleting consensus metadata
I20260927 21:07:13.214428 2660 raft_consensus.cc:2229] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:13.214501 2660 raft_consensus.cc:2258] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:13.214494 2100 catalog_manager.cc:4841] TS dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871): tablet 5afcb477119e4d329a428b27f00c9169 (table test-table [id=9e1c992bab6042e983d0805b2d8a4cd5]) successfully deleted
I20260927 21:07:13.214087 2252 tablet_service.cc:1467] Processing CreateTablet for tablet a59c938868d04bd6a4e92541391bf851 (DEFAULT_TABLE table=test-table [id=961a4e3b2b2147e596ad37a062548647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:13.214663 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59c938868d04bd6a4e92541391bf851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:13.214740 2618 tablet_bootstrap.cc:492] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: No bootstrap required, opened a new log
I20260927 21:07:13.214792 2618 ts_tablet_manager.cc:1397] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:07:13.214926 2618 raft_consensus.cc:348] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.214990 2618 raft_consensus.cc:374] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.215010 2618 raft_consensus.cc:729] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d338e474ae46188de9b34d3eaf97b0, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.215056 2618 consensus_queue.cc:260] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.215184 2618 ts_tablet_manager.cc:1428] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:13.215992 2619 tablet_bootstrap.cc:492] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Bootstrap starting.
I20260927 21:07:13.216315 2619 tablet_bootstrap.cc:654] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:13.216961 2619 tablet_bootstrap.cc:492] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: No bootstrap required, opened a new log
I20260927 21:07:13.217010 2659 raft_consensus.cc:2258] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:13.217015 2619 ts_tablet_manager.cc:1397] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:13.217214 2619 raft_consensus.cc:348] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.217289 2619 raft_consensus.cc:374] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.217317 2619 raft_consensus.cc:729] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.217356 2619 consensus_queue.cc:260] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.217420 2660 ts_tablet_manager.cc:1905] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:13.217438 2619 ts_tablet_manager.cc:1428] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:13.218878 2660 ts_tablet_manager.cc:1918] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:13.218954 2660 log.cc:1196] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/5afcb477119e4d329a428b27f00c9169
I20260927 21:07:13.219197 2659 ts_tablet_manager.cc:1905] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:13.219259 2660 ts_tablet_manager.cc:1939] T 5afcb477119e4d329a428b27f00c9169 P 42d338e474ae46188de9b34d3eaf97b0: Deleting consensus metadata
I20260927 21:07:13.219491 2622 tablet_bootstrap.cc:492] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: No bootstrap required, opened a new log
I20260927 21:07:13.219542 2622 ts_tablet_manager.cc:1397] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 21:07:13.219671 2622 raft_consensus.cc:348] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.219736 2622 raft_consensus.cc:374] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:13.219758 2622 raft_consensus.cc:729] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc03939d49ca43ffa216cf3d2450bb07, State: Initialized, Role: FOLLOWER
I20260927 21:07:13.219801 2622 consensus_queue.cc:260] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.219895 2622 ts_tablet_manager.cc:1428] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:13.220337 2102 catalog_manager.cc:4841] TS 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999): tablet 5afcb477119e4d329a428b27f00c9169 (table test-table [id=9e1c992bab6042e983d0805b2d8a4cd5]) successfully deleted
I20260927 21:07:13.220779 2659 ts_tablet_manager.cc:1918] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:13.220832 2659 log.cc:1196] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/5afcb477119e4d329a428b27f00c9169
I20260927 21:07:13.220996 2659 ts_tablet_manager.cc:1939] T 5afcb477119e4d329a428b27f00c9169 P ed0191a61f194b03976d39be8c89ef21: Deleting consensus metadata
I20260927 21:07:13.221304 2099 catalog_manager.cc:4841] TS ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037): tablet 5afcb477119e4d329a428b27f00c9169 (table test-table [id=9e1c992bab6042e983d0805b2d8a4cd5]) successfully deleted
W20260927 21:07:13.240319 2599 tablet.cc:2369] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:13.394510 2626 raft_consensus.cc:482] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:13.394618 2626 raft_consensus.cc:504] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.394975 2626 leader_election.cc:288] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037), 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:13.398393 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.395248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.98.2:37859)
0927 21:07:13.395330 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.395340 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.395428 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.395599 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.395600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.395603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.395679 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.395681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.396318 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.396319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.397528 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.397530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.398109 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.398110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.398188 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.398191 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.398268 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.398308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":38,"threads_started":1}
I20260927 21:07:13.398384 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.395200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37859 (local address 127.0.98.3:44999)
0927 21:07:13.395272 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.395274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.395405 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:07:13.395523 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.395524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.395533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.395584 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.395773 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.396256 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.397614 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.398027 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.398108 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.398118 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.398272 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.398323 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:07:13.398634 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.395189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38321 (local address 127.0.98.1:34037)
0927 21:07:13.395231 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.395234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.395380 (+ 146us) server_negotiation.cc:407] Connection header received
0927 21:07:13.395447 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.395448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.395457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.395511 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:13.396334 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.396751 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.397728 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.398289 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.398351 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.398360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.398539 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.398569 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:13.398872 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.395159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.2:38321)
0927 21:07:13.395316 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.395327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.395394 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.395627 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.395627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.395629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:13.395684 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.395685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.396832 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.396833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.397631 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.397633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.398392 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.398393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.398465 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.398469 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.398755 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.398791 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260927 21:07:13.398885 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c938868d04bd6a4e92541391bf851" candidate_uuid: "dc03939d49ca43ffa216cf3d2450bb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0" is_pre_election: true
I20260927 21:07:13.398981 2552 raft_consensus.cc:2454] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc03939d49ca43ffa216cf3d2450bb07 in term 0.
I20260927 21:07:13.399068 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c938868d04bd6a4e92541391bf851" candidate_uuid: "dc03939d49ca43ffa216cf3d2450bb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0191a61f194b03976d39be8c89ef21" is_pre_election: true
I20260927 21:07:13.399199 2272 raft_consensus.cc:2454] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc03939d49ca43ffa216cf3d2450bb07 in term 0.
I20260927 21:07:13.399201 2346 leader_election.cc:302] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, dc03939d49ca43ffa216cf3d2450bb07; no voters:
I20260927 21:07:13.399329 2626 raft_consensus.cc:2781] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:13.399371 2626 raft_consensus.cc:482] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:13.399397 2626 raft_consensus.cc:3037] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.400063 2626 raft_consensus.cc:504] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.400202 2626 leader_election.cc:288] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [CANDIDATE]: Term 1 election: Requested vote from peers ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037), 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:13.400346 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c938868d04bd6a4e92541391bf851" candidate_uuid: "dc03939d49ca43ffa216cf3d2450bb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0191a61f194b03976d39be8c89ef21"
I20260927 21:07:13.400441 2272 raft_consensus.cc:3037] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.401185 2272 raft_consensus.cc:2454] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc03939d49ca43ffa216cf3d2450bb07 in term 1.
I20260927 21:07:13.401351 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59c938868d04bd6a4e92541391bf851" candidate_uuid: "dc03939d49ca43ffa216cf3d2450bb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0"
I20260927 21:07:13.401451 2552 raft_consensus.cc:3037] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:13.401999 2552 raft_consensus.cc:2454] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc03939d49ca43ffa216cf3d2450bb07 in term 1.
I20260927 21:07:13.402125 2343 leader_election.cc:302] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc03939d49ca43ffa216cf3d2450bb07, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:13.402220 2626 raft_consensus.cc:2781] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:13.402338 2626 raft_consensus.cc:686] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 LEADER]: Becoming Leader. State: Replica: dc03939d49ca43ffa216cf3d2450bb07, State: Running, Role: LEADER
I20260927 21:07:13.402405 2626 consensus_queue.cc:237] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:13.403066 2113 catalog_manager.cc:5495] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "dc03939d49ca43ffa216cf3d2450bb07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:13.444116 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.441582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.0.98.2:42871)
0927 21:07:13.441650 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.441652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.441660 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:13.441772 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.441773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.441795 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.441884 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:13.442015 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.442499 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.443290 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.443500 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.443594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.443765 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.443884 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.443959 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.444002 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.444024 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:07:13.449349 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.446493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57011 (local address 127.0.98.1:34037)
0927 21:07:13.446563 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.446565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.446653 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:07:13.446713 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.446714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.446725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.446780 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:13.446909 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.447499 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.448653 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.448861 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.448944 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.449102 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.449141 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.449201 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.449259 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.449277 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:07:13.449566 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.446406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.2:57011)
0927 21:07:13.446575 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.446586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.446660 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.446784 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.446785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.446786 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:13.446845 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.446846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.447581 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.447583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.448555 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.448557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.448943 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.448944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.449010 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.449021 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.449201 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.449203 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.449255 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.449296 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260927 21:07:13.451628 2552 raft_consensus.cc:1261] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Refusing update from remote peer dc03939d49ca43ffa216cf3d2450bb07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:13.451870 2272 raft_consensus.cc:1261] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Refusing update from remote peer dc03939d49ca43ffa216cf3d2450bb07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:13.451898 2667 consensus_queue.cc:1035] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:13.452085 2667 consensus_queue.cc:1035] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:13.458559 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.456493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37135 (local address 127.0.98.2:42871)
0927 21:07:13.456531 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:13.456533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:13.456541 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:13.456594 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:13.456594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:13.456605 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:13.456664 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:13.456774 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.457335 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.457999 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.458164 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.458214 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.458315 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.458345 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.458394 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:13.458455 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:07:13.458473 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:13.458566 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:13.456413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42871 (local address 127.0.98.1:37135)
0927 21:07:13.456489 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:13.456501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:13.456542 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:13.456676 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:13.456677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:13.456678 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:13.456716 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.456717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.457391 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.457392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.457934 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:13.457936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:13.458212 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:13.458213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:13.458245 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:13.458248 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:13.458406 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:13.458407 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:13.458452 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:07:13.458491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 21:07:13.458870 2644 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:07:14.978185 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:14.978291 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:14.979125 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Sending DeleteTablet for 3 replicas of tablet a59c938868d04bd6a4e92541391bf851
I20260927 21:07:14.979378 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet a59c938868d04bd6a4e92541391bf851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44502
I20260927 21:07:14.979365 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet a59c938868d04bd6a4e92541391bf851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 21:07:14.979422 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet a59c938868d04bd6a4e92541391bf851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36944
I20260927 21:07:14.979552 2697 tablet_replica.cc:330] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: stopping tablet replica
I20260927 21:07:14.979595 2697 raft_consensus.cc:2229] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:14.979630 2697 raft_consensus.cc:2258] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:14.979645 2699 tablet_replica.cc:330] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: stopping tablet replica
I20260927 21:07:14.979692 2699 raft_consensus.cc:2229] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:14.979766 2699 raft_consensus.cc:2258] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:14.979945 2697 ts_tablet_manager.cc:1905] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:14.981899 2697 ts_tablet_manager.cc:1918] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:07:14.981966 2697 log.cc:1196] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/a59c938868d04bd6a4e92541391bf851
I20260927 21:07:14.982170 2699 ts_tablet_manager.cc:1905] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:14.982197 2697 ts_tablet_manager.cc:1939] T a59c938868d04bd6a4e92541391bf851 P ed0191a61f194b03976d39be8c89ef21: Deleting consensus metadata
I20260927 21:07:14.982641 2099 catalog_manager.cc:4841] TS ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037): tablet a59c938868d04bd6a4e92541391bf851 (table test-table [id=961a4e3b2b2147e596ad37a062548647]) successfully deleted
I20260927 21:07:14.982666 2698 tablet_replica.cc:330] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: stopping tablet replica
I20260927 21:07:14.982726 2698 raft_consensus.cc:2229] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:14.982837 2698 raft_consensus.cc:2258] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:14.983070 2698 ts_tablet_manager.cc:1905] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:14.983747 2699 ts_tablet_manager.cc:1918] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:07:14.983817 2699 log.cc:1196] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/a59c938868d04bd6a4e92541391bf851
I20260927 21:07:14.984089 2699 ts_tablet_manager.cc:1939] T a59c938868d04bd6a4e92541391bf851 P 42d338e474ae46188de9b34d3eaf97b0: Deleting consensus metadata
I20260927 21:07:14.984402 2102 catalog_manager.cc:4841] TS 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999): tablet a59c938868d04bd6a4e92541391bf851 (table test-table [id=961a4e3b2b2147e596ad37a062548647]) successfully deleted
I20260927 21:07:14.984953 2698 ts_tablet_manager.cc:1918] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:07:14.985018 2698 log.cc:1196] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/a59c938868d04bd6a4e92541391bf851
I20260927 21:07:14.985251 2698 ts_tablet_manager.cc:1939] T a59c938868d04bd6a4e92541391bf851 P dc03939d49ca43ffa216cf3d2450bb07: Deleting consensus metadata
I20260927 21:07:14.985596 2100 catalog_manager.cc:4841] TS dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871): tablet a59c938868d04bd6a4e92541391bf851 (table test-table [id=961a4e3b2b2147e596ad37a062548647]) successfully deleted
Sep 27 21:07:14 dist-test-slave-8bgs krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:07:15 dist-test-slave-8bgs krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:15.013022 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:14.997108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.0.98.62:45043)
0927 21:07:14.997242 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:14.997244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:14.997266 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:07:14.997298 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:14.997298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:14.997307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:14.997372 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:14.998036 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:14.998577 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:14.999448 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:14.999639 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:14.999687 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.010353 (+ 10666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:15.010355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:15.010355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:15.010364 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:15.011960 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:15.012321 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:15.012321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:15.012322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:15.012377 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:15.012608 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:15.012608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:15.012609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:15.012710 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:15.012791 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.012885 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.012949 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260927 21:07:15.014493 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:15.014669 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:15.016031 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 43bb18d1c214475880dda550f0aad88e (DEFAULT_TABLE table=test-table [id=8964190867414e38bb0ef76a5a368773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:15.016039 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 43bb18d1c214475880dda550f0aad88e (DEFAULT_TABLE table=test-table [id=8964190867414e38bb0ef76a5a368773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:15.016140 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 43bb18d1c214475880dda550f0aad88e (DEFAULT_TABLE table=test-table [id=8964190867414e38bb0ef76a5a368773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:15.016176 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bb18d1c214475880dda550f0aad88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:15.016176 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bb18d1c214475880dda550f0aad88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:15.016255 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bb18d1c214475880dda550f0aad88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:15.017588 2709 tablet_bootstrap.cc:492] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Bootstrap starting.
I20260927 21:07:15.017611 2710 tablet_bootstrap.cc:492] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Bootstrap starting.
I20260927 21:07:15.017817 2708 tablet_bootstrap.cc:492] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Bootstrap starting.
I20260927 21:07:15.018040 2709 tablet_bootstrap.cc:654] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:15.018052 2710 tablet_bootstrap.cc:654] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:15.018187 2708 tablet_bootstrap.cc:654] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:15.018656 2710 tablet_bootstrap.cc:492] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: No bootstrap required, opened a new log
I20260927 21:07:15.018708 2710 ts_tablet_manager.cc:1397] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:15.018730 2709 tablet_bootstrap.cc:492] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: No bootstrap required, opened a new log
I20260927 21:07:15.018765 2708 tablet_bootstrap.cc:492] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: No bootstrap required, opened a new log
I20260927 21:07:15.018774 2709 ts_tablet_manager.cc:1397] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:15.018863 2708 ts_tablet_manager.cc:1397] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:15.018838 2710 raft_consensus.cc:348] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.018906 2710 raft_consensus.cc:374] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:15.018936 2710 raft_consensus.cc:729] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc03939d49ca43ffa216cf3d2450bb07, State: Initialized, Role: FOLLOWER
I20260927 21:07:15.018908 2709 raft_consensus.cc:348] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.018970 2709 raft_consensus.cc:374] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:15.018978 2710 consensus_queue.cc:260] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.019002 2709 raft_consensus.cc:729] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d338e474ae46188de9b34d3eaf97b0, State: Initialized, Role: FOLLOWER
I20260927 21:07:15.018976 2708 raft_consensus.cc:348] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.019059 2708 raft_consensus.cc:374] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:15.019044 2709 consensus_queue.cc:260] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.019107 2708 raft_consensus.cc:729] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Initialized, Role: FOLLOWER
I20260927 21:07:15.019073 2710 ts_tablet_manager.cc:1428] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:15.019178 2708 consensus_queue.cc:260] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.019235 2709 ts_tablet_manager.cc:1428] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:15.019722 2708 ts_tablet_manager.cc:1428] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:07:15.071436 2714 raft_consensus.cc:482] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:15.071537 2714 raft_consensus.cc:504] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.071852 2714 leader_election.cc:288] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037), dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871)
I20260927 21:07:15.075208 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.072092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46601 (local address 127.0.98.1:34037)
0927 21:07:15.072259 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:15.072261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:15.072329 (+ 68us) server_negotiation.cc:407] Connection header received
0927 21:07:15.072404 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:15.072405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:15.072418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:15.072506 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:15.072682 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.073263 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.074432 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.074818 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.074901 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.074913 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.075075 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.075138 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260927 21:07:15.075208 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.072057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.3:46601)
0927 21:07:15.072272 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:15.072282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:15.072348 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:15.072509 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:15.072510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:15.072513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:15.072592 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.072594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.073341 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.073343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.074349 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.074352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.074899 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.074900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.074973 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.074977 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:15.075064 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:07:15.075126 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":3,"thread_start_us":134,"threads_started":1}
I20260927 21:07:15.075306 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.072108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54789 (local address 127.0.98.2:42871)
0927 21:07:15.072304 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:15.072306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:15.072337 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:07:15.072432 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:15.072433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:15.072443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:15.072526 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:15.072660 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.073299 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.074505 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.074886 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.074979 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.074991 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.075178 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.075222 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260927 21:07:15.075624 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.072057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42871 (local address 127.0.98.3:54789)
0927 21:07:15.072271 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:15.072283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:15.072348 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:15.072528 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:15.072529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:15.072531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:15.072588 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.072591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.073387 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.073388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.074420 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.074423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.074983 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.074984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.075045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.075049 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:15.075167 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 21:07:15.075209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":135,"threads_started":1}
I20260927 21:07:15.075609 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bb18d1c214475880dda550f0aad88e" candidate_uuid: "42d338e474ae46188de9b34d3eaf97b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0191a61f194b03976d39be8c89ef21" is_pre_election: true
I20260927 21:07:15.075721 2272 raft_consensus.cc:2454] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42d338e474ae46188de9b34d3eaf97b0 in term 0.
I20260927 21:07:15.075911 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bb18d1c214475880dda550f0aad88e" candidate_uuid: "42d338e474ae46188de9b34d3eaf97b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07" is_pre_election: true
I20260927 21:07:15.075963 2483 leader_election.cc:302] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:15.076026 2411 raft_consensus.cc:2454] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42d338e474ae46188de9b34d3eaf97b0 in term 0.
I20260927 21:07:15.076084 2714 raft_consensus.cc:2781] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:15.076128 2714 raft_consensus.cc:482] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:15.076156 2714 raft_consensus.cc:3037] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:15.076880 2714 raft_consensus.cc:504] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.077013 2714 leader_election.cc:288] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [CANDIDATE]: Term 1 election: Requested vote from peers ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037), dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871)
I20260927 21:07:15.077158 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bb18d1c214475880dda550f0aad88e" candidate_uuid: "42d338e474ae46188de9b34d3eaf97b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07"
I20260927 21:07:15.077153 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bb18d1c214475880dda550f0aad88e" candidate_uuid: "42d338e474ae46188de9b34d3eaf97b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0191a61f194b03976d39be8c89ef21"
I20260927 21:07:15.077234 2411 raft_consensus.cc:3037] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:15.077248 2272 raft_consensus.cc:3037] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:15.077859 2411 raft_consensus.cc:2454] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42d338e474ae46188de9b34d3eaf97b0 in term 1.
I20260927 21:07:15.077864 2272 raft_consensus.cc:2454] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42d338e474ae46188de9b34d3eaf97b0 in term 1.
I20260927 21:07:15.078023 2484 leader_election.cc:302] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, dc03939d49ca43ffa216cf3d2450bb07; no voters:
I20260927 21:07:15.078119 2714 raft_consensus.cc:2781] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:15.078248 2714 raft_consensus.cc:686] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 1 LEADER]: Becoming Leader. State: Replica: 42d338e474ae46188de9b34d3eaf97b0, State: Running, Role: LEADER
I20260927 21:07:15.078333 2714 consensus_queue.cc:237] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.079025 2114 catalog_manager.cc:5495] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "42d338e474ae46188de9b34d3eaf97b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: HEALTHY } } }
I20260927 21:07:15.095432 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.092697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.0.98.3:44999)
0927 21:07:15.092831 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:15.092833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:15.092839 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:07:15.092870 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:15.092870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:15.092888 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:15.092977 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:15.093128 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.093718 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.094495 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.094731 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.094807 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.095012 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:15.095183 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:15.095279 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.095317 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.095336 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260927 21:07:15.100046 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.097837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37001 (local address 127.0.98.1:34037)
0927 21:07:15.097879 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:15.097881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:15.097918 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:07:15.098003 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:15.098003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:15.098048 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:15.098111 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:15.098248 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.098696 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.099471 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.099659 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.099712 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.099854 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:15.099878 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:15.099917 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.099969 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.099985 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:15.100062 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.097724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.98.3:37001)
0927 21:07:15.097864 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:15.097874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:15.097925 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:15.098113 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:15.098114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:15.098115 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:15.098162 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.098163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.098765 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.098766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.099403 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.099405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.099723 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.099724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.099776 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.099787 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:15.099930 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:15.099932 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:15.099961 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 21:07:15.099992 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260927 21:07:15.101964 2411 raft_consensus.cc:1261] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Refusing update from remote peer 42d338e474ae46188de9b34d3eaf97b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:15.101972 2272 raft_consensus.cc:1261] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Refusing update from remote peer 42d338e474ae46188de9b34d3eaf97b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:15.102205 2714 consensus_queue.cc:1035] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:15.102375 2719 consensus_queue.cc:1035] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:15.108803 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.106638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42949 (local address 127.0.98.3:44999)
0927 21:07:15.106676 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:15.106678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:15.106766 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:07:15.106834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:15.106834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:15.106842 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:15.106881 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:15.106950 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.107388 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.108083 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.108364 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.108436 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.108589 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:15.108620 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:15.108675 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.108725 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.108744 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:15.108853 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.106595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44999 (local address 127.0.98.1:42949)
0927 21:07:15.106708 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:15.106719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:15.106778 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:15.106882 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:15.106883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:15.106884 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:15.106917 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.106918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.107441 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.107442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.107991 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:15.107992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.108458 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.108459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:15.108525 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.108530 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:15.108677 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:15.108679 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:15.108723 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:07:15.108775 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":77,"threads_started":1}
I20260927 21:07:15.114926 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:15.115021 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:15.115592 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Sending DeleteTablet for 3 replicas of tablet 43bb18d1c214475880dda550f0aad88e
I20260927 21:07:15.115854 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bb18d1c214475880dda550f0aad88e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 21:07:15.115895 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bb18d1c214475880dda550f0aad88e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36944
I20260927 21:07:15.115943 2697 tablet_replica.cc:330] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: stopping tablet replica
I20260927 21:07:15.115976 2699 tablet_replica.cc:330] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: stopping tablet replica
I20260927 21:07:15.116000 2697 raft_consensus.cc:2229] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:15.116015 2699 raft_consensus.cc:2229] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:15.116039 2697 raft_consensus.cc:2258] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:15.116098 2699 raft_consensus.cc:2258] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:15.116156 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:15.116285 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:15.116346 2699 ts_tablet_manager.cc:1905] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:15.116405 2697 ts_tablet_manager.cc:1905] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:15.117450 2532 tablet_service.cc:1467] Processing CreateTablet for tablet b21ffcbb579f4e9ca1a01e47b3af7d01 (DEFAULT_TABLE table=test-table [id=7ca2207cab8f42c5804b0aa7a7b987ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:15.117550 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21ffcbb579f4e9ca1a01e47b3af7d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:15.118261 2699 ts_tablet_manager.cc:1918] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:15.118325 2699 log.cc:1196] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/43bb18d1c214475880dda550f0aad88e
I20260927 21:07:15.118542 2699 ts_tablet_manager.cc:1939] T 43bb18d1c214475880dda550f0aad88e P 42d338e474ae46188de9b34d3eaf97b0: Deleting consensus metadata
I20260927 21:07:15.118825 2709 tablet_bootstrap.cc:492] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Bootstrap starting.
I20260927 21:07:15.118830 2102 catalog_manager.cc:4841] TS 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999): tablet 43bb18d1c214475880dda550f0aad88e (table test-table [id=8964190867414e38bb0ef76a5a368773]) successfully deleted
I20260927 21:07:15.119237 2709 tablet_bootstrap.cc:654] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:15.119311 2252 tablet_service.cc:1467] Processing CreateTablet for tablet b21ffcbb579f4e9ca1a01e47b3af7d01 (DEFAULT_TABLE table=test-table [id=7ca2207cab8f42c5804b0aa7a7b987ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:15.119526 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bb18d1c214475880dda550f0aad88e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44502
I20260927 21:07:15.119599 2391 tablet_service.cc:1467] Processing CreateTablet for tablet b21ffcbb579f4e9ca1a01e47b3af7d01 (DEFAULT_TABLE table=test-table [id=7ca2207cab8f42c5804b0aa7a7b987ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:15.119620 2697 ts_tablet_manager.cc:1918] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:15.119668 2697 log.cc:1196] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/43bb18d1c214475880dda550f0aad88e
I20260927 21:07:15.119702 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21ffcbb579f4e9ca1a01e47b3af7d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:15.119884 2697 ts_tablet_manager.cc:1939] T 43bb18d1c214475880dda550f0aad88e P ed0191a61f194b03976d39be8c89ef21: Deleting consensus metadata
I20260927 21:07:15.119617 2698 tablet_replica.cc:330] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: stopping tablet replica
I20260927 21:07:15.120808 2710 tablet_bootstrap.cc:492] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Bootstrap starting.
I20260927 21:07:15.120837 2698 raft_consensus.cc:2229] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:15.120869 2698 raft_consensus.cc:2258] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:15.120970 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21ffcbb579f4e9ca1a01e47b3af7d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:15.121126 2710 tablet_bootstrap.cc:654] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:15.121155 2698 ts_tablet_manager.cc:1905] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:15.121923 2710 tablet_bootstrap.cc:492] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: No bootstrap required, opened a new log
I20260927 21:07:15.121977 2710 ts_tablet_manager.cc:1397] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:15.122088 2710 raft_consensus.cc:348] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.122145 2710 raft_consensus.cc:374] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:15.122164 2710 raft_consensus.cc:729] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc03939d49ca43ffa216cf3d2450bb07, State: Initialized, Role: FOLLOWER
I20260927 21:07:15.122205 2710 consensus_queue.cc:260] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.122293 2710 ts_tablet_manager.cc:1428] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:15.122413 2708 tablet_bootstrap.cc:492] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21: Bootstrap starting.
I20260927 21:07:15.122529 2698 ts_tablet_manager.cc:1918] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:15.122598 2698 log.cc:1196] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/43bb18d1c214475880dda550f0aad88e
I20260927 21:07:15.122699 2708 tablet_bootstrap.cc:654] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:15.122826 2698 ts_tablet_manager.cc:1939] T 43bb18d1c214475880dda550f0aad88e P dc03939d49ca43ffa216cf3d2450bb07: Deleting consensus metadata
I20260927 21:07:15.123172 2099 catalog_manager.cc:4841] TS ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037): tablet 43bb18d1c214475880dda550f0aad88e (table test-table [id=8964190867414e38bb0ef76a5a368773]) successfully deleted
I20260927 21:07:15.123282 2100 catalog_manager.cc:4841] TS dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871): tablet 43bb18d1c214475880dda550f0aad88e (table test-table [id=8964190867414e38bb0ef76a5a368773]) successfully deleted
I20260927 21:07:15.123673 2709 tablet_bootstrap.cc:492] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: No bootstrap required, opened a new log
I20260927 21:07:15.123720 2709 ts_tablet_manager.cc:1397] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:07:15.123896 2709 raft_consensus.cc:348] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.123968 2709 raft_consensus.cc:374] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:15.123993 2709 raft_consensus.cc:729] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d338e474ae46188de9b34d3eaf97b0, State: Initialized, Role: FOLLOWER
I20260927 21:07:15.124047 2709 consensus_queue.cc:260] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.124150 2709 ts_tablet_manager.cc:1428] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:15.124501 2708 tablet_bootstrap.cc:492] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21: No bootstrap required, opened a new log
I20260927 21:07:15.124544 2708 ts_tablet_manager.cc:1397] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:15.124655 2708 raft_consensus.cc:348] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.124724 2708 raft_consensus.cc:374] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:15.124754 2708 raft_consensus.cc:729] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Initialized, Role: FOLLOWER
I20260927 21:07:15.124797 2708 consensus_queue.cc:260] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.124902 2708 ts_tablet_manager.cc:1428] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:15.215402 2683 raft_consensus.cc:482] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:15.215489 2683 raft_consensus.cc:504] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.215679 2683 leader_election.cc:288] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871), 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:15.215899 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21ffcbb579f4e9ca1a01e47b3af7d01" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0" is_pre_election: true
I20260927 21:07:15.215894 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21ffcbb579f4e9ca1a01e47b3af7d01" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07" is_pre_election: true
I20260927 21:07:15.216005 2552 raft_consensus.cc:2454] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 0.
I20260927 21:07:15.216007 2411 raft_consensus.cc:2454] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 0.
I20260927 21:07:15.216183 2206 leader_election.cc:302] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:15.216275 2683 raft_consensus.cc:2781] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:15.216325 2683 raft_consensus.cc:482] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:15.216344 2683 raft_consensus.cc:3037] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:15.217255 2683 raft_consensus.cc:504] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.217386 2683 leader_election.cc:288] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 election: Requested vote from peers dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871), 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999)
I20260927 21:07:15.217607 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21ffcbb579f4e9ca1a01e47b3af7d01" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d338e474ae46188de9b34d3eaf97b0"
I20260927 21:07:15.217689 2552 raft_consensus.cc:3037] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:15.217649 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21ffcbb579f4e9ca1a01e47b3af7d01" candidate_uuid: "ed0191a61f194b03976d39be8c89ef21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc03939d49ca43ffa216cf3d2450bb07"
I20260927 21:07:15.217758 2411 raft_consensus.cc:3037] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:15.218391 2411 raft_consensus.cc:2454] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 1.
I20260927 21:07:15.218395 2552 raft_consensus.cc:2454] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed0191a61f194b03976d39be8c89ef21 in term 1.
I20260927 21:07:15.218663 2206 leader_election.cc:302] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d338e474ae46188de9b34d3eaf97b0, ed0191a61f194b03976d39be8c89ef21; no voters:
I20260927 21:07:15.218750 2683 raft_consensus.cc:2781] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:15.218797 2683 raft_consensus.cc:686] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 1 LEADER]: Becoming Leader. State: Replica: ed0191a61f194b03976d39be8c89ef21, State: Running, Role: LEADER
I20260927 21:07:15.218845 2683 consensus_queue.cc:237] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } }
I20260927 21:07:15.219456 2113 catalog_manager.cc:5495] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed0191a61f194b03976d39be8c89ef21 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ed0191a61f194b03976d39be8c89ef21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed0191a61f194b03976d39be8c89ef21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:15.241146 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:15.238390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.98.1:34037)
0927 21:07:15.239213 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:15.239214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:15.239221 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:15.239271 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:15.239272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:15.239283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:15.239341 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:15.239463 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.239914 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.240595 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:15.240766 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:15.240815 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:15.240919 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:15.240942 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:15.240975 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:15.241028 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:07:15.241043 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795}
I20260927 21:07:15.243072 2411 raft_consensus.cc:1261] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Refusing update from remote peer ed0191a61f194b03976d39be8c89ef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:15.243073 2552 raft_consensus.cc:1261] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Refusing update from remote peer ed0191a61f194b03976d39be8c89ef21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:15.243336 2678 consensus_queue.cc:1035] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc03939d49ca43ffa216cf3d2450bb07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:15.243557 2678 consensus_queue.cc:1035] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d338e474ae46188de9b34d3eaf97b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:07:15.247326 2676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:07:16.764014 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:16.764132 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:16.764904 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62a33c7912974fd0900c72984e0d924c: Sending DeleteTablet for 3 replicas of tablet b21ffcbb579f4e9ca1a01e47b3af7d01
I20260927 21:07:16.765147 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet b21ffcbb579f4e9ca1a01e47b3af7d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33274
I20260927 21:07:16.765202 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet b21ffcbb579f4e9ca1a01e47b3af7d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44502
I20260927 21:07:16.765183 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet b21ffcbb579f4e9ca1a01e47b3af7d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36944
I20260927 21:07:16.765375 2759 tablet_replica.cc:330] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21: stopping tablet replica
I20260927 21:07:16.765395 2760 tablet_replica.cc:330] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: stopping tablet replica
I20260927 21:07:16.765446 2760 raft_consensus.cc:2229] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:16.765435 2759 raft_consensus.cc:2229] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:16.765482 2760 raft_consensus.cc:2258] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:16.765506 2759 raft_consensus.cc:2258] T b21ffcbb579f4e9ca1a01e47b3af7d01 P ed0191a61f194b03976d39be8c89ef21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:16.765497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 2187
I20260927 21:07:16.766297 2760 ts_tablet_manager.cc:1905] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:16.768316 2760 ts_tablet_manager.cc:1918] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:07:16.768373 2760 log.cc:1196] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/b21ffcbb579f4e9ca1a01e47b3af7d01
I20260927 21:07:16.768589 2760 ts_tablet_manager.cc:1939] T b21ffcbb579f4e9ca1a01e47b3af7d01 P dc03939d49ca43ffa216cf3d2450bb07: Deleting consensus metadata
I20260927 21:07:16.769028 2100 catalog_manager.cc:4841] TS dc03939d49ca43ffa216cf3d2450bb07 (127.0.98.2:42871): tablet b21ffcbb579f4e9ca1a01e47b3af7d01 (table test-table [id=7ca2207cab8f42c5804b0aa7a7b987ab]) successfully deleted
I20260927 21:07:16.771220 2761 tablet_replica.cc:330] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: stopping tablet replica
I20260927 21:07:16.771286 2761 raft_consensus.cc:2229] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:16.771330 2761 raft_consensus.cc:2258] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:16.771534 2761 ts_tablet_manager.cc:1905] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:16.772758 2761 ts_tablet_manager.cc:1918] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:07:16.772807 2761 log.cc:1196] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/b21ffcbb579f4e9ca1a01e47b3af7d01
I20260927 21:07:16.773013 2761 ts_tablet_manager.cc:1939] T b21ffcbb579f4e9ca1a01e47b3af7d01 P 42d338e474ae46188de9b34d3eaf97b0: Deleting consensus metadata
I20260927 21:07:16.773377 2102 catalog_manager.cc:4841] TS 42d338e474ae46188de9b34d3eaf97b0 (127.0.98.3:44999): tablet b21ffcbb579f4e9ca1a01e47b3af7d01 (table test-table [id=7ca2207cab8f42c5804b0aa7a7b987ab]) successfully deleted
W20260927 21:07:16.773686 2099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34037 (error 108)
I20260927 21:07:16.773782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 2327
I20260927 21:07:16.774791 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:16.774586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34037 (local address 127.0.0.1:51754)
0927 21:07:16.774704 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:16.774758 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":76,"threads_started":1}
W20260927 21:07:16.774897 2099 catalog_manager.cc:4584] TS ed0191a61f194b03976d39be8c89ef21 (127.0.98.1:34037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b21ffcbb579f4e9ca1a01e47b3af7d01: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34037: connect: Connection refused (error 111)
I20260927 21:07:16.779735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 2467
I20260927 21:07:16.785609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 2083
2026-09-27T21:07:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5530 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:16 dist-test-slave-8bgs krb5kdc[2766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:16 dist-test-slave-8bgs krb5kdc[2766](info): set up 2 sockets
Sep 27 21:07:16 dist-test-slave-8bgs krb5kdc[2766](info): commencing operation
krb5kdc: starting...
W20260927 21:07:17.818812 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:17 dist-test-slave-8bgs krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:17.927815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:18.011600 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:18.011819 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:18.011859 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:18.013005 2779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:18.013067 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:18.013093 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:18.013121 2779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:18.013152 2779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:18.014631 2779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:18.014899 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:18.015136 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:18.019773 2784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:18.019779 2785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:18.019992 2779 server_base.cc:1034] running on GCE node
W20260927 21:07:18.019757 2787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:18.020311 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:18.020598 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:18.021756 2779 hybrid_clock.cc:648] HybridClock initialized: now 1790543238021754 us; error 41 us; skew 500 ppm
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:18.024165 2779 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:18.024597 2779 webserver.cc:460] Webserver started at http://127.0.98.62:41277/ using document root <none> and password file <none>
I20260927 21:07:18.024786 2779 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:18.024828 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:18.024915 2779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:18.025885 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "1a65c088159544458ef13f271d80d53c"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "b332c9f178eddeb3d61ded6f7586614a"
server_key_iv: "64a0f0c4c3994f42bed9b9870485c707"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.026180 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a65c088159544458ef13f271d80d53c"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "b332c9f178eddeb3d61ded6f7586614a"
server_key_iv: "64a0f0c4c3994f42bed9b9870485c707"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.027453 2779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:18.028129 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.028323 2779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:18.028399 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "1a65c088159544458ef13f271d80d53c"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "b332c9f178eddeb3d61ded6f7586614a"
server_key_iv: "64a0f0c4c3994f42bed9b9870485c707"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.028484 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:18.044646 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:18.046484 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:18.046631 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:18.050639 2779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44457
I20260927 21:07:18.050699 2846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44457 every 8 connection(s)
I20260927 21:07:18.051199 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:18.051916 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:18.052484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 2779
I20260927 21:07:18.052582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:18.052757 392 external_mini_cluster.cc:1428] Setting key 9918e3db52c7f499fc37c7455fac4b60
I20260927 21:07:18.054977 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c: Bootstrap starting.
I20260927 21:07:18.055589 2847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:18.055893 2847 log.cc:824] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:18.056623 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c: No bootstrap required, opened a new log
I20260927 21:07:18.058734 2847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a65c088159544458ef13f271d80d53c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44457 } }
I20260927 21:07:18.058890 2847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:18.058944 2847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a65c088159544458ef13f271d80d53c, State: Initialized, Role: FOLLOWER
I20260927 21:07:18.059120 2847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a65c088159544458ef13f271d80d53c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44457 } }
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:18.059222 2847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:18.059268 2847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:18.059316 2847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:18.059805 2847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a65c088159544458ef13f271d80d53c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44457 } }
I20260927 21:07:18.059919 2847 leader_election.cc:302] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a65c088159544458ef13f271d80d53c; no voters:
I20260927 21:07:18.060088 2847 leader_election.cc:288] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:18.060134 2852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:18.060335 2852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [term 1 LEADER]: Becoming Leader. State: Replica: 1a65c088159544458ef13f271d80d53c, State: Running, Role: LEADER
I20260927 21:07:18.060371 2847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:18.060498 2852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a65c088159544458ef13f271d80d53c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44457 } }
I20260927 21:07:18.060798 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a65c088159544458ef13f271d80d53c. Latest consensus state: current_term: 1 leader_uuid: "1a65c088159544458ef13f271d80d53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a65c088159544458ef13f271d80d53c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44457 } } }
I20260927 21:07:18.060860 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:18.060827 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a65c088159544458ef13f271d80d53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a65c088159544458ef13f271d80d53c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44457 } } }
I20260927 21:07:18.060923 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:18.061057 2859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:18.061733 2859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:18.063370 2859 catalog_manager.cc:1347] Generated new cluster ID: 56fc44f523524ee1a99596615fae524c
I20260927 21:07:18.063421 2859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:18.063983 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.053299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.0.98.62:44457)
0927 21:07:18.053453 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:18.053461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:18.053485 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:07:18.053873 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:18.053879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:18.053922 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:18.054035 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:18.054832 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.055804 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.056817 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.057208 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.057301 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.059711 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:18.059715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:18.059715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:18.059726 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:18.062200 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.062593 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.062595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.062596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.062715 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.062949 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.062951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.062951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.063266 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:18.063408 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:18.063671 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 21:07:18.063802 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260927 21:07:18.071745 2859 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:18.072440 2859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:18.076517 2859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a65c088159544458ef13f271d80d53c: Generated new TSK 0
I20260927 21:07:18.076668 2859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:18.104144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44457
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:18.183856 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:18.184077 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:18.184113 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:18.185273 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:18.185335 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:18.185379 2873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:18.186867 2873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:18.187176 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:18.187408 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:18.192202 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:18.192183 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:18.192260 2873 server_base.cc:1034] running on GCE node
W20260927 21:07:18.192178 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:18.192587 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:18.192823 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:18.193977 2873 hybrid_clock.cc:648] HybridClock initialized: now 1790543238193974 us; error 33 us; skew 500 ppm
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:18.196470 2873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:18.196880 2873 webserver.cc:460] Webserver started at http://127.0.98.1:46479/ using document root <none> and password file <none>
I20260927 21:07:18.197104 2873 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:18.197170 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:18.197309 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:18.198316 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed58513fd85c439d8aabc1ca00929bdd"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "2159b218585062c4af1bcbf83865a3c2"
server_key_iv: "c5bc63edf3d032e689b2a59b50c988a8"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.198659 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed58513fd85c439d8aabc1ca00929bdd"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "2159b218585062c4af1bcbf83865a3c2"
server_key_iv: "c5bc63edf3d032e689b2a59b50c988a8"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.200001 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:18.200650 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.200814 2873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:18.200904 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "ed58513fd85c439d8aabc1ca00929bdd"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "2159b218585062c4af1bcbf83865a3c2"
server_key_iv: "c5bc63edf3d032e689b2a59b50c988a8"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.200976 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:18.218245 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:18.220093 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:18.220252 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:18.220458 2873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:18.220767 2873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:18.220817 2873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.220858 2873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:18.220884 2873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.226394 2873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41031
I20260927 21:07:18.226471 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41031 every 8 connection(s)
I20260927 21:07:18.226785 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:18.228922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 2873
I20260927 21:07:18.229017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:18.229198 392 external_mini_cluster.cc:1428] Setting key 0b739832727a48ee8531e1d2124f89e8
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:18.236825 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.227560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56349 (local address 127.0.98.62:44457)
0927 21:07:18.227623 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:18.227625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:18.227810 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:07:18.228707 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:18.228709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:18.228718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:18.228790 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:18.229588 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.230085 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.231298 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.231493 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.231551 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.233970 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:18.233972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:18.233972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:18.233979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:18.235431 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.235851 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.235852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.235853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.235916 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.236163 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.236164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.236165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.236282 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:18.236355 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:18.236646 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 21:07:18.236736 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:07:18.237018 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.227533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44457 (local address 127.0.98.1:56349)
0927 21:07:18.227686 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:18.227726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:18.228623 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:18.228865 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:18.228869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:18.229250 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:18.229492 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:18.229498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.230177 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.230178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:18.231223 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:18.231226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.231569 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.231570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:18.231714 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.232264 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:18.232274 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:18.233868 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:18.235524 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:18.235531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:18.235533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:18.235761 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:18.235984 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:18.235984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:18.235985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:18.236097 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:18.236358 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:18.236388 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:18.236517 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 21:07:18.236625 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:07:18.236776 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260927 21:07:18.237486 3002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44457
I20260927 21:07:18.237563 3002 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:18.237762 3002 heartbeater.cc:511] Master 127.0.98.62:44457 requested a full tablet report, sending...
I20260927 21:07:18.238273 2811 ts_manager.cc:194] Registered new tserver with Master: ed58513fd85c439d8aabc1ca00929bdd (127.0.98.1:41031)
I20260927 21:07:18.238837 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:18.251009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44457
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:18.330922 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:18.331182 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:18.331210 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:18.332430 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:18.332471 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:18.332500 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:18.333911 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:18.334102 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:18.334271 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:18.339147 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:18.339164 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:18.339169 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:18.339382 3007 server_base.cc:1034] running on GCE node
I20260927 21:07:18.339617 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:18.339866 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:18.341025 3007 hybrid_clock.cc:648] HybridClock initialized: now 1790543238341023 us; error 36 us; skew 500 ppm
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:18.343423 3007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:18.343868 3007 webserver.cc:460] Webserver started at http://127.0.98.2:44735/ using document root <none> and password file <none>
I20260927 21:07:18.344055 3007 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:18.344094 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:18.344178 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:18.345144 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "81a2dd066a92418aa56f3826ddd78f8e"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "1eb5dc35a037f9091edd0488dc1be8c3"
server_key_iv: "49e8535f427534049aab357591175b51"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.345429 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "81a2dd066a92418aa56f3826ddd78f8e"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "1eb5dc35a037f9091edd0488dc1be8c3"
server_key_iv: "49e8535f427534049aab357591175b51"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.346624 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:18.347283 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.347448 3007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:18.347524 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "81a2dd066a92418aa56f3826ddd78f8e"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "1eb5dc35a037f9091edd0488dc1be8c3"
server_key_iv: "49e8535f427534049aab357591175b51"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.347575 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:18.362123 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:18.364123 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:18.364238 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:18.364415 3007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:18.364777 3007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:18.364845 3007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.364892 3007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:18.364912 3007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.373202 3007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43553
I20260927 21:07:18.373272 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43553 every 8 connection(s)
I20260927 21:07:18.373698 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:18.375316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3007
I20260927 21:07:18.375437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:18.375640 392 external_mini_cluster.cc:1428] Setting key 349ff61f8a1dd32334f72ea2f631c2e9
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:18.386049 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.374661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50027 (local address 127.0.98.62:44457)
0927 21:07:18.374733 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:18.374735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:18.374982 (+ 247us) server_negotiation.cc:407] Connection header received
0927 21:07:18.376627 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:18.376629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:18.376641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:18.376713 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:18.377678 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.378311 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.379466 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.379674 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.379752 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.382423 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:18.382425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:18.382425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:18.382433 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:18.384377 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.384873 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.384874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.384875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.384933 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.385246 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.385247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.385248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.385388 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:18.385471 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:18.385853 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:07:18.385942 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:07:18.386612 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.374649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44457 (local address 127.0.98.2:50027)
0927 21:07:18.374844 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:18.374892 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:18.376499 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:18.376863 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:18.376867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:18.377271 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:18.377558 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:18.377565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.378405 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.378406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:18.379372 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:18.379374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.379756 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.379757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:18.379902 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.380463 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:18.380475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:18.382324 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:18.384487 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:18.384494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:18.384497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:18.384785 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:18.385003 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:18.385004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:18.385004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:18.385152 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:18.385502 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:18.385509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:18.385676 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:07:18.386177 (+ 501us) client_negotiation.cc:240] Negotiation successful
0927 21:07:18.386360 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260927 21:07:18.387039 3136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44457
I20260927 21:07:18.387171 3136 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:18.387394 3136 heartbeater.cc:511] Master 127.0.98.62:44457 requested a full tablet report, sending...
I20260927 21:07:18.387830 2811 ts_manager.cc:194] Registered new tserver with Master: 81a2dd066a92418aa56f3826ddd78f8e (127.0.98.2:43553)
I20260927 21:07:18.388397 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:18.409305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44457
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:18.492233 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:18.492460 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:18.492497 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:18.493621 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:18.493688 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:18.493731 3141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:18.495329 3141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:18.495573 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:18.495797 3141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:18.500429 3147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:18.500442 3146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:18.500452 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:18.500557 3141 server_base.cc:1034] running on GCE node
I20260927 21:07:18.500873 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:18.501118 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:18.502254 3141 hybrid_clock.cc:648] HybridClock initialized: now 1790543238502242 us; error 27 us; skew 500 ppm
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:18.504755 3141 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:18.505204 3141 webserver.cc:460] Webserver started at http://127.0.98.3:34061/ using document root <none> and password file <none>
I20260927 21:07:18.505425 3141 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:18.505468 3141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:18.505627 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:18.506600 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6b6cbc59a52470f992309aac7343597"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "93ceb387f93b9b1d94352df16190af7b"
server_key_iv: "23cb926892d2adc0b831fb3f9c186b12"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.506966 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6b6cbc59a52470f992309aac7343597"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "93ceb387f93b9b1d94352df16190af7b"
server_key_iv: "23cb926892d2adc0b831fb3f9c186b12"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.508219 3141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:18.508893 3156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.509050 3141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.509147 3141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "b6b6cbc59a52470f992309aac7343597"
format_stamp: "Formatted at 2026-09-27 21:07:18 on dist-test-slave-8bgs"
server_key: "93ceb387f93b9b1d94352df16190af7b"
server_key_iv: "23cb926892d2adc0b831fb3f9c186b12"
server_key_version: "encryptionkey@0"
I20260927 21:07:18.509222 3141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:18.521389 3141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:18.523618 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:18.523780 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:18.523998 3141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:18.524314 3141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:18.524369 3141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.524402 3141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:18.524430 3141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:18.530153 3141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36915
I20260927 21:07:18.530196 3269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36915 every 8 connection(s)
I20260927 21:07:18.530483 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:18.534360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3141
I20260927 21:07:18.534480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:18.534675 392 external_mini_cluster.cc:1428] Setting key b9e499add311b137be1f07db4bba8551
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:18.539836 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.531196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47167 (local address 127.0.98.62:44457)
0927 21:07:18.531254 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:18.531256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:18.531457 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:07:18.532537 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:18.532539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:18.532551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:18.532615 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:18.533236 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.533755 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.534863 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.535212 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.535283 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.537274 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:18.537275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:18.537275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:18.537282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:18.538647 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.538972 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.538973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.538973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.539020 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:18.539332 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:18.539333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:18.539333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:18.539439 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:18.539491 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:18.539719 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 21:07:18.539775 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:07:18.540055 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:18.531197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44457 (local address 127.0.98.3:47167)
0927 21:07:18.531338 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:18.531371 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:18.532430 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:18.532692 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:18.532696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:18.532949 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:18.533130 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:18.533135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.533852 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.533853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:18.534779 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:18.534782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:18.535299 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:18.535300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:18.535431 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:18.535906 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:18.535914 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:18.537195 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:18.538713 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:18.538717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:18.538719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:18.538912 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:18.539077 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:18.539078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:18.539078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:18.539253 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:18.539494 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:18.539498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:18.539612 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 21:07:18.539697 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:07:18.539810 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260927 21:07:18.540467 3270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44457
I20260927 21:07:18.540561 3270 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:18.540750 3270 heartbeater.cc:511] Master 127.0.98.62:44457 requested a full tablet report, sending...
I20260927 21:07:18.541107 2811 ts_manager.cc:194] Registered new tserver with Master: b6b6cbc59a52470f992309aac7343597 (127.0.98.3:36915)
I20260927 21:07:18.541471 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47167
I20260927 21:07:18.546294 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:07:18.556545 2850 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:18.555760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.98.62:44457)
0927 21:07:18.555816 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:18.555817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:18.555898 (+ 81us) server_negotiation.cc:407] Connection header received
0927 21:07:18.555946 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:18.555947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:18.555955 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:18.556008 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:18.556497 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55638: BlockingRecv error: recv got EOF from 127.0.0.1:55638 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:07:18.557551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 2873
I20260927 21:07:18.562202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3007
I20260927 21:07:18.566234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3141
I20260927 21:07:18.570299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 2779
2026-09-27T21:07:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1782 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[3282](info): set up 2 sockets
Sep 27 21:07:18 dist-test-slave-8bgs krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260927 21:07:19.594892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:19 dist-test-slave-8bgs krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:19.705447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:19.782536 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:19.782728 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:19.782749 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:19.783891 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:19.783931 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:19.783946 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:19.783957 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:19.784008 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:19.785426 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:19.785660 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:19.785871 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:19.790530 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:19.790611 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:19.790616 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:19.790630 3295 server_base.cc:1034] running on GCE node
I20260927 21:07:19.790983 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:19.791278 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:19.792436 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790543239792422 us; error 28 us; skew 500 ppm
Sep 27 21:07:19 dist-test-slave-8bgs krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:19.795437 3295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:19.795908 3295 webserver.cc:460] Webserver started at http://127.0.98.62:32885/ using document root <none> and password file <none>
I20260927 21:07:19.796151 3295 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:19.796197 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:19.796351 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:19.797401 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "4de0a79bf47d48e09b23f27ba87fcb66"
format_stamp: "Formatted at 2026-09-27 21:07:19 on dist-test-slave-8bgs"
server_key: "02ed4311f5bf76b45e6c0265fe831d58"
server_key_iv: "f33631383e526c442b594872198bf323"
server_key_version: "encryptionkey@0"
I20260927 21:07:19.797765 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4de0a79bf47d48e09b23f27ba87fcb66"
format_stamp: "Formatted at 2026-09-27 21:07:19 on dist-test-slave-8bgs"
server_key: "02ed4311f5bf76b45e6c0265fe831d58"
server_key_iv: "f33631383e526c442b594872198bf323"
server_key_version: "encryptionkey@0"
I20260927 21:07:19.799031 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:07:19.799844 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:19.800037 3295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:19.800128 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "4de0a79bf47d48e09b23f27ba87fcb66"
format_stamp: "Formatted at 2026-09-27 21:07:19 on dist-test-slave-8bgs"
server_key: "02ed4311f5bf76b45e6c0265fe831d58"
server_key_iv: "f33631383e526c442b594872198bf323"
server_key_version: "encryptionkey@0"
I20260927 21:07:19.800199 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:19.812073 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:19.813776 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:19.813925 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:19.817876 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45361
I20260927 21:07:19.817950 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45361 every 8 connection(s)
I20260927 21:07:19.818252 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:19.819039 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:19.819890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3295
I20260927 21:07:19.819981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:19.820158 392 external_mini_cluster.cc:1428] Setting key 28c7693bdf955c9e7446284fd4a93772
I20260927 21:07:19.822172 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66: Bootstrap starting.
I20260927 21:07:19.822763 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:19.823040 3363 log.cc:824] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:19.823726 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66: No bootstrap required, opened a new log
I20260927 21:07:19.825589 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260927 21:07:19.825754 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:19.825819 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de0a79bf47d48e09b23f27ba87fcb66, State: Initialized, Role: FOLLOWER
I20260927 21:07:19.825968 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260927 21:07:19.826049 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:19.826092 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:19.826138 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 21:07:19 dist-test-slave-8bgs krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:19.826690 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260927 21:07:19.826840 3363 leader_election.cc:302] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de0a79bf47d48e09b23f27ba87fcb66; no voters:
I20260927 21:07:19.827025 3363 leader_election.cc:288] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:19.827121 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:19.827293 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [term 1 LEADER]: Becoming Leader. State: Replica: 4de0a79bf47d48e09b23f27ba87fcb66, State: Running, Role: LEADER
I20260927 21:07:19.827317 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:19.827418 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } }
I20260927 21:07:19.827716 3368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de0a79bf47d48e09b23f27ba87fcb66. Latest consensus state: current_term: 1 leader_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } } }
I20260927 21:07:19.827880 3368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:19.827729 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de0a79bf47d48e09b23f27ba87fcb66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45361 } } }
I20260927 21:07:19.828140 3377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:19.828145 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:19.828680 3377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:19.830199 3377 catalog_manager.cc:1347] Generated new cluster ID: 833fad442d7b4fcfb06b56016128c1df
I20260927 21:07:19.830257 3377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:19.830776 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:19.820648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.62:45361)
0927 21:07:19.820820 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:19.820829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:19.820861 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:07:19.821254 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:19.821260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:19.821315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:19.821424 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:19.822212 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:19.823217 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:19.824046 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:19.824471 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:19.824589 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:19.826943 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:19.826948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:19.826948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:19.826961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:19.829167 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:19.829561 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:19.829563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:19.829564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:19.829665 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:19.829906 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:19.829907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:19.829907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:19.830198 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:19.830332 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:19.830459 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:07:19.830588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260927 21:07:19.835553 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:07:19.836750 3377 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:19.837468 3377 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:19.846189 3377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de0a79bf47d48e09b23f27ba87fcb66: Generated new TSK 0
I20260927 21:07:19.846379 3377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:19.855123 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:19.851393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.98.62:45361)
0927 21:07:19.851438 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:19.851440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:19.851543 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:07:19.851575 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:19.851576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:19.851587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:19.851661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:19.852013 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:19.852802 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:19.853568 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:19.853903 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:19.853972 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:19.854641 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:19.854642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:19.854642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:19.854860 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:19.854877 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:19.854883 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:19.854943 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:19.854992 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:07:19.855018 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:19.857551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3295
2026-09-27T21:07:19Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1285 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:19 dist-test-slave-8bgs krb5kdc[3396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:19 dist-test-slave-8bgs krb5kdc[3396](info): set up 2 sockets
Sep 27 21:07:19 dist-test-slave-8bgs krb5kdc[3396](info): commencing operation
krb5kdc: starting...
W20260927 21:07:20.878966 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:20 dist-test-slave-8bgs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:20.993670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:21.072454 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:21.072696 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:21.072736 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:21.073859 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:21.073927 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:21.073951 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:21.073982 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:21.074011 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:21.075580 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:21.075855 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:21.076076 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:21.080844 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.080849 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.080924 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:21.081095 3409 server_base.cc:1034] running on GCE node
I20260927 21:07:21.081316 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:21.081545 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:21.082679 3409 hybrid_clock.cc:648] HybridClock initialized: now 1790543241082654 us; error 39 us; skew 500 ppm
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:21.092574 3409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:21.093048 3409 webserver.cc:460] Webserver started at http://127.0.98.62:38061/ using document root <none> and password file <none>
I20260927 21:07:21.093246 3409 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:21.093289 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:21.093376 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:21.094368 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "a716ace9969e4350b43b2bce4badf2f3"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "eeefa8522adacec374f9f6ea50964863"
server_key_iv: "2e775fcbf0fcf85271d5fd3a74cdaee2"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.094735 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a716ace9969e4350b43b2bce4badf2f3"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "eeefa8522adacec374f9f6ea50964863"
server_key_iv: "2e775fcbf0fcf85271d5fd3a74cdaee2"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.096088 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:21.096784 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.096997 3409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:21.097095 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "a716ace9969e4350b43b2bce4badf2f3"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "eeefa8522adacec374f9f6ea50964863"
server_key_iv: "2e775fcbf0fcf85271d5fd3a74cdaee2"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.097149 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:21.114588 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:21.116601 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:21.116712 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:21.120378 3409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43177
I20260927 21:07:21.120429 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43177 every 8 connection(s)
I20260927 21:07:21.120779 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:21.121744 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:21.124046 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3: Bootstrap starting.
I20260927 21:07:21.124614 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:21.124905 3477 log.cc:824] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:21.125581 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3: No bootstrap required, opened a new log
I20260927 21:07:21.127184 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a716ace9969e4350b43b2bce4badf2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43177 } }
I20260927 21:07:21.127351 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:21.127398 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a716ace9969e4350b43b2bce4badf2f3, State: Initialized, Role: FOLLOWER
I20260927 21:07:21.127538 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a716ace9969e4350b43b2bce4badf2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43177 } }
I20260927 21:07:21.127619 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:21.127664 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:21.127713 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:21.127894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3409
I20260927 21:07:21.127969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:21.128139 392 external_mini_cluster.cc:1428] Setting key c4c5827800f0e4e95ed3dcc07abc6249
I20260927 21:07:21.128788 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a716ace9969e4350b43b2bce4badf2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43177 } }
I20260927 21:07:21.128906 3477 leader_election.cc:302] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a716ace9969e4350b43b2bce4badf2f3; no voters:
I20260927 21:07:21.129067 3477 leader_election.cc:288] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:21.129089 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:21.129194 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [term 1 LEADER]: Becoming Leader. State: Replica: a716ace9969e4350b43b2bce4badf2f3, State: Running, Role: LEADER
I20260927 21:07:21.129290 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a716ace9969e4350b43b2bce4badf2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43177 } }
I20260927 21:07:21.129318 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:21.129592 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a716ace9969e4350b43b2bce4badf2f3. Latest consensus state: current_term: 1 leader_uuid: "a716ace9969e4350b43b2bce4badf2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a716ace9969e4350b43b2bce4badf2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43177 } } }
I20260927 21:07:21.129593 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a716ace9969e4350b43b2bce4badf2f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a716ace9969e4350b43b2bce4badf2f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43177 } } }
I20260927 21:07:21.129699 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:21.129732 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:21.130391 3488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:21.131001 3488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:21.132593 3488 catalog_manager.cc:1347] Generated new cluster ID: df9671ca9b494d5fa752d21a568122cc
I20260927 21:07:21.132647 3488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:21.139539 3488 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:21.140120 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.128639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.0.98.62:43177)
0927 21:07:21.128794 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:21.128801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:21.128825 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:07:21.129032 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:21.129037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:21.129081 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:21.129279 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:21.130107 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.131420 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.132443 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.133031 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.133136 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.136085 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:21.136089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:21.136090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:21.136101 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:21.138018 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.138436 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.138439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.138439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.138558 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.138793 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.138794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.138794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.139039 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:21.139214 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:21.139781 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 21:07:21.139915 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260927 21:07:21.140209 3488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:21.142875 3488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a716ace9969e4350b43b2bce4badf2f3: Generated new TSK 0
I20260927 21:07:21.143045 3488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:21.170269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43177
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:21.251314 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:21.251530 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:21.251567 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:21.252712 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:21.252777 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:21.252821 3503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:21.254220 3503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:21.254469 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:21.254693 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:21.260263 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.260274 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.260272 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:21.260560 3503 server_base.cc:1034] running on GCE node
I20260927 21:07:21.260748 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:21.261016 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:21.262172 3503 hybrid_clock.cc:648] HybridClock initialized: now 1790543241262161 us; error 30 us; skew 500 ppm
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:21.264604 3503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:21.265060 3503 webserver.cc:460] Webserver started at http://127.0.98.1:37631/ using document root <none> and password file <none>
I20260927 21:07:21.265291 3503 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:21.265357 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:21.265512 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:21.266608 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4a65694bd454633bef4e352412d820e"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "fa93b0937bae0d2c5c794edd43fa8d0d"
server_key_iv: "aa8f015c11129f3fcc74c34c5f7bcbdd"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.266985 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4a65694bd454633bef4e352412d820e"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "fa93b0937bae0d2c5c794edd43fa8d0d"
server_key_iv: "aa8f015c11129f3fcc74c34c5f7bcbdd"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.268291 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:21.268978 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.269152 3503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.269250 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "e4a65694bd454633bef4e352412d820e"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "fa93b0937bae0d2c5c794edd43fa8d0d"
server_key_iv: "aa8f015c11129f3fcc74c34c5f7bcbdd"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.269335 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:21.312409 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:21.314268 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:21.314435 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:21.314689 3503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:21.315001 3503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:21.315052 3503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.315117 3503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:21.315165 3503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.320591 3503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33653
I20260927 21:07:21.320634 3631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33653 every 8 connection(s)
I20260927 21:07:21.320943 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:21.326334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3503
I20260927 21:07:21.326445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:21.326634 392 external_mini_cluster.cc:1428] Setting key d0b99ab951842706765364f769d0a727
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:21.330358 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.321573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44083 (local address 127.0.98.62:43177)
0927 21:07:21.321614 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:21.321616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:21.321849 (+ 233us) server_negotiation.cc:407] Connection header received
0927 21:07:21.322317 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:21.322318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:21.322327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:21.322384 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:21.323333 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.323863 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.324803 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.324993 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.325059 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.327416 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:21.327418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:21.327419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:21.327426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:21.328905 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.329312 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.329314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.329314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.329370 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.329676 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.329677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.329678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.329841 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:21.329911 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:21.330233 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 21:07:21.330298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:21.330690 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.321548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43177 (local address 127.0.98.1:44083)
0927 21:07:21.321716 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:21.321757 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:21.322241 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:21.322450 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:21.322454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:21.323009 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:21.323228 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:21.323233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.323944 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.323945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:21.324721 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:21.324723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.325075 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.325076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:21.325171 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.325687 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:21.325703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:21.327346 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:21.328997 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:21.329004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:21.329006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:21.329228 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:21.329453 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:21.329454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:21.329454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:21.329611 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:21.329919 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:21.329925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:21.330092 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:07:21.330206 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:07:21.330379 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260927 21:07:21.331295 3632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43177
I20260927 21:07:21.331398 3632 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:21.331586 3632 heartbeater.cc:511] Master 127.0.98.62:43177 requested a full tablet report, sending...
I20260927 21:07:21.332154 3441 ts_manager.cc:194] Registered new tserver with Master: e4a65694bd454633bef4e352412d820e (127.0.98.1:33653)
I20260927 21:07:21.333155 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44083
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:21.350948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43177
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:21.430698 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:21.430894 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:21.430917 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:21.432029 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:21.432063 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:21.432089 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:21.433411 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:21.433591 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:21.433753 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:21.438591 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:21.438699 3637 server_base.cc:1034] running on GCE node
W20260927 21:07:21.438591 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.438620 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:21.438990 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:21.439257 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:21.440416 3637 hybrid_clock.cc:648] HybridClock initialized: now 1790543241440411 us; error 34 us; skew 500 ppm
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:21.442759 3637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:21.443207 3637 webserver.cc:460] Webserver started at http://127.0.98.2:32917/ using document root <none> and password file <none>
I20260927 21:07:21.443392 3637 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:21.443431 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:21.443529 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:21.444489 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "4878f8c142b54dc89b4e51db6903f473"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "eb7d368aee369981f2ad6a9f59a474f1"
server_key_iv: "925691e70d1820908d3ea383a2123f2f"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.444792 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4878f8c142b54dc89b4e51db6903f473"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "eb7d368aee369981f2ad6a9f59a474f1"
server_key_iv: "925691e70d1820908d3ea383a2123f2f"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.446002 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:21.446657 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.446830 3637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:21.446918 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "4878f8c142b54dc89b4e51db6903f473"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "eb7d368aee369981f2ad6a9f59a474f1"
server_key_iv: "925691e70d1820908d3ea383a2123f2f"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.446993 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:21.458946 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:21.461261 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:21.461422 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:21.461649 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:21.461952 3637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:21.461985 3637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.462029 3637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:21.462056 3637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.467902 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41653
I20260927 21:07:21.467976 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41653 every 8 connection(s)
I20260927 21:07:21.468300 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:21.475571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3637
I20260927 21:07:21.475689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:21.475914 392 external_mini_cluster.cc:1428] Setting key c1571ca0c41cb3abd88740b5738e5edb
I20260927 21:07:21.477926 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.468999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55711 (local address 127.0.98.62:43177)
0927 21:07:21.469045 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:21.469048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:21.469251 (+ 203us) server_negotiation.cc:407] Connection header received
0927 21:07:21.469789 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:21.469791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:21.469808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:21.469856 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:21.470810 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.471353 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.472411 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.472604 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.472701 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.474826 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:21.474827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:21.474827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:21.474834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:21.476406 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.476896 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.476897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.476898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.476956 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.477199 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.477200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.477201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.477314 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:21.477390 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:21.477713 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:07:21.477806 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:21.478449 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.468960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43177 (local address 127.0.98.2:55711)
0927 21:07:21.469098 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:21.469145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:21.469679 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:21.469937 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:21.469943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:21.470513 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:21.470705 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:21.470709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.471442 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.471444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:21.472324 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:21.472326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.472717 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.472718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:21.472832 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.473406 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:21.473415 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:21.474704 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:21.476587 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:21.476592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:21.476593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:21.476804 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:21.477020 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:21.477021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:21.477022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:21.477130 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:21.477393 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:21.477399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:21.477533 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:07:21.478024 (+ 491us) client_negotiation.cc:240] Negotiation successful
0927 21:07:21.478201 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260927 21:07:21.478983 3766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43177
I20260927 21:07:21.479075 3766 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:21.479290 3766 heartbeater.cc:511] Master 127.0.98.62:43177 requested a full tablet report, sending...
I20260927 21:07:21.479646 3441 ts_manager.cc:194] Registered new tserver with Master: 4878f8c142b54dc89b4e51db6903f473 (127.0.98.2:41653)
I20260927 21:07:21.480098 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55711
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:21.499291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43177
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:21.580808 3771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:21.581035 3771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:21.581073 3771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:21.582188 3771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:21.582252 3771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:21.582293 3771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:21.583711 3771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:21.583961 3771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:21.584227 3771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:21.589035 3779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.589051 3777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:21.589044 3776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:21.589265 3771 server_base.cc:1034] running on GCE node
I20260927 21:07:21.589481 3771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:21.589745 3771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:21.590881 3771 hybrid_clock.cc:648] HybridClock initialized: now 1790543241590872 us; error 26 us; skew 500 ppm
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:21.593324 3771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:21.593739 3771 webserver.cc:460] Webserver started at http://127.0.98.3:34573/ using document root <none> and password file <none>
I20260927 21:07:21.593968 3771 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:21.594035 3771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:21.594185 3771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:21.595331 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "4898c5057b24494a9a47a40e68cbf5ef"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "777bc24dace81de6435577055ef214fb"
server_key_iv: "fa9537beee98937b59ea54b72800b9a2"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.595690 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4898c5057b24494a9a47a40e68cbf5ef"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "777bc24dace81de6435577055ef214fb"
server_key_iv: "fa9537beee98937b59ea54b72800b9a2"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.597453 3771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:21.598212 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.598383 3771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:21.598480 3771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "4898c5057b24494a9a47a40e68cbf5ef"
format_stamp: "Formatted at 2026-09-27 21:07:21 on dist-test-slave-8bgs"
server_key: "777bc24dace81de6435577055ef214fb"
server_key_iv: "fa9537beee98937b59ea54b72800b9a2"
server_key_version: "encryptionkey@0"
I20260927 21:07:21.598560 3771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:21.612567 3771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:21.614290 3771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:21.614450 3771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:21.614674 3771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:21.614991 3771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:21.615044 3771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.615078 3771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:21.615173 3771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:21.620724 3771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46743
I20260927 21:07:21.620760 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46743 every 8 connection(s)
I20260927 21:07:21.621099 3771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:21.624722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3771
I20260927 21:07:21.624815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:21.625046 392 external_mini_cluster.cc:1428] Setting key 5d51e86786c237cc697f5d2f74d83ed1
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:21.630007 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.621852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57801 (local address 127.0.98.62:43177)
0927 21:07:21.621924 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:21.621925 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:21.622102 (+ 177us) server_negotiation.cc:407] Connection header received
0927 21:07:21.622560 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:21.622561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:21.622569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:21.622623 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:21.623582 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.624056 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.625068 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.625309 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.625397 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.627485 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:21.627487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:21.627487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:21.627494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:21.628608 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.629000 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.629001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.629002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.629058 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.629361 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.629362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.629362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.629487 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:21.629567 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:21.629842 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 21:07:21.629924 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:07:21.630182 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.621829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43177 (local address 127.0.98.3:57801)
0927 21:07:21.621973 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:21.622013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:21.622481 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:21.622675 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:21.622678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:21.623299 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:21.623502 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:21.623507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.624130 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.624131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:21.624987 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:21.624989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.625412 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.625413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:21.625535 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.626039 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:21.626047 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:21.627366 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:21.628709 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:21.628715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:21.628717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:21.628926 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:21.629122 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:21.629123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:21.629124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:21.629287 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:21.629571 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:21.629576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:21.629710 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:07:21.629809 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:07:21.629951 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260927 21:07:21.630666 3900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43177
I20260927 21:07:21.630739 3900 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:21.630903 3900 heartbeater.cc:511] Master 127.0.98.62:43177 requested a full tablet report, sending...
I20260927 21:07:21.631294 3441 ts_manager.cc:194] Registered new tserver with Master: 4898c5057b24494a9a47a40e68cbf5ef (127.0.98.3:46743)
I20260927 21:07:21.631690 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57801
I20260927 21:07:21.636741 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543241, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:21.658720 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:21.651900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.0.98.62:43177)
0927 21:07:21.652002 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:21.652005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:21.652109 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:07:21.652221 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:21.652222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:21.652233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:21.652307 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:21.652881 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.653363 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.654116 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:21.654313 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:21.654396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:21.656626 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:21.656627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:21.656628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:21.656634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:21.657762 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.658066 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.658067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.658067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.658113 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:21.658272 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:21.658273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:21.658273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:21.658402 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:21.658472 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:21.658599 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:07:21.658656 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 21:07:21.658962 3441 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48776
I20260927 21:07:21.659411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3503
I20260927 21:07:21.663643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3637
I20260927 21:07:21.667737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3771
I20260927 21:07:21.671914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3409
2026-09-27T21:07:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1816 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3912](info): set up 2 sockets
Sep 27 21:07:21 dist-test-slave-8bgs krb5kdc[3912](info): commencing operation
krb5kdc: starting...
W20260927 21:07:22.687027 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:22 dist-test-slave-8bgs krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:22.793481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42701
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:22.871652 3925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:22.871881 3925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:22.871917 3925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:22.873063 3925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:22.873133 3925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:22.873160 3925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:22.873207 3925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:22.873236 3925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:22.874704 3925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:22.874964 3925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:22.875242 3925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:22.879860 3933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:22.879848 3930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:22.879878 3925 server_base.cc:1034] running on GCE node
W20260927 21:07:22.879864 3931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:22.880328 3925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:22.880602 3925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:22.881767 3925 hybrid_clock.cc:648] HybridClock initialized: now 1790543242881754 us; error 28 us; skew 500 ppm
Sep 27 21:07:22 dist-test-slave-8bgs krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:22.884111 3925 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:22.884512 3925 webserver.cc:460] Webserver started at http://127.0.98.62:37315/ using document root <none> and password file <none>
I20260927 21:07:22.884739 3925 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:22.884805 3925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:22.884954 3925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:22.885931 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529"
format_stamp: "Formatted at 2026-09-27 21:07:22 on dist-test-slave-8bgs"
server_key: "9f93e67604ac35807759abe6f131e463"
server_key_iv: "48d6bce29b270d56cfcd131c4b5d9b57"
server_key_version: "encryptionkey@0"
I20260927 21:07:22.886274 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529"
format_stamp: "Formatted at 2026-09-27 21:07:22 on dist-test-slave-8bgs"
server_key: "9f93e67604ac35807759abe6f131e463"
server_key_iv: "48d6bce29b270d56cfcd131c4b5d9b57"
server_key_version: "encryptionkey@0"
I20260927 21:07:22.887570 3925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:22.888234 3940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:22.888393 3925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:22.888489 3925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529"
format_stamp: "Formatted at 2026-09-27 21:07:22 on dist-test-slave-8bgs"
server_key: "9f93e67604ac35807759abe6f131e463"
server_key_iv: "48d6bce29b270d56cfcd131c4b5d9b57"
server_key_version: "encryptionkey@0"
I20260927 21:07:22.888563 3925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:22.906747 3925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:22.908538 3925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:22.908699 3925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:22.912380 3925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42701
I20260927 21:07:22.912429 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42701 every 8 connection(s)
I20260927 21:07:22.912791 3925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:22.913622 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:22.915865 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529: Bootstrap starting.
I20260927 21:07:22.916415 3993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:22.916704 3993 log.cc:824] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:22.917320 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529: No bootstrap required, opened a new log
I20260927 21:07:22.917691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 3925
I20260927 21:07:22.917820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:22.918004 392 external_mini_cluster.cc:1428] Setting key b5b9cc5c2e861faa5d7381ccdb1bce49
I20260927 21:07:22.919313 3993 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:07:22.919482 3993 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:22.919567 3993 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f1e9b88a9a49a0aae876b9f6fd4529, State: Initialized, Role: FOLLOWER
I20260927 21:07:22.919719 3993 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:07:22.919816 3993 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:22.919847 3993 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:22.919904 3993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:22.920861 3993 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:07:22.920946 3993 leader_election.cc:302] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f1e9b88a9a49a0aae876b9f6fd4529; no voters:
I20260927 21:07:22.921109 3993 leader_election.cc:288] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:22.921145 3998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:22.921290 3998 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [term 1 LEADER]: Becoming Leader. State: Replica: f6f1e9b88a9a49a0aae876b9f6fd4529, State: Running, Role: LEADER
I20260927 21:07:22.921360 3993 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:22.921471 3998 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } }
I20260927 21:07:22.921834 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f1e9b88a9a49a0aae876b9f6fd4529. Latest consensus state: current_term: 1 leader_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } } }
I20260927 21:07:22.921995 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:22.922405 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:22.922987 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:22.923127 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f1e9b88a9a49a0aae876b9f6fd4529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42701 } } }
I20260927 21:07:22.923233 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:22.924687 4007 catalog_manager.cc:1347] Generated new cluster ID: d75eb28aacfc44f38b0885cb0e222c1d
I20260927 21:07:22.924746 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:22 dist-test-slave-8bgs krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:22.929942 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:22.918495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.0.98.62:42701)
0927 21:07:22.918658 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:22.918666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:22.918689 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:07:22.919283 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:22.919290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:22.919334 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:22.919441 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:22.920315 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:22.921377 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:22.922472 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:22.923131 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:22.923239 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:22.926333 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:22.926337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:22.926337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:22.926348 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:22.928241 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:22.928629 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:22.928632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:22.928632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:22.928746 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:22.929019 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:22.929020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:22.929020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:22.929292 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:22.929419 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:22.929627 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 21:07:22.929758 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260927 21:07:22.938184 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:22.938671 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:22.942276 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f1e9b88a9a49a0aae876b9f6fd4529: Generated new TSK 0
I20260927 21:07:22.942436 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:22.970397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:23.052434 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:23.052661 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:23.052696 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:23.053779 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:23.053846 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:23.053910 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:23.055428 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:23.055686 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:23.055917 4019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:23.060758 4024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:23.060716 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:23.060716 4025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:23.060990 4019 server_base.cc:1034] running on GCE node
I20260927 21:07:23.061242 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:23.061465 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:23.062598 4019 hybrid_clock.cc:648] HybridClock initialized: now 1790543243062570 us; error 42 us; skew 500 ppm
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:23.065116 4019 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:23.065510 4019 webserver.cc:460] Webserver started at http://127.0.98.1:38409/ using document root <none> and password file <none>
I20260927 21:07:23.065733 4019 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:23.065840 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:23.065975 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:23.066975 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9838b64f96a48b2bb18ed2c3f844f47"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "b43d830641dca4bbf1f3b863ddb11cb0"
server_key_iv: "37abdc2ec0fcc2e1757505fe8f600056"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.067380 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9838b64f96a48b2bb18ed2c3f844f47"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "b43d830641dca4bbf1f3b863ddb11cb0"
server_key_iv: "37abdc2ec0fcc2e1757505fe8f600056"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.068601 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:23.069336 4034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.069499 4019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:23.069587 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "d9838b64f96a48b2bb18ed2c3f844f47"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "b43d830641dca4bbf1f3b863ddb11cb0"
server_key_iv: "37abdc2ec0fcc2e1757505fe8f600056"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.069669 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:23.092243 4019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:23.094444 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:23.094592 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:23.094806 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:23.095137 4019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:23.095197 4019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.095237 4019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:23.095264 4019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.100818 4019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44209
I20260927 21:07:23.100895 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44209 every 8 connection(s)
I20260927 21:07:23.101202 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:23.106462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4019
I20260927 21:07:23.106571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:23.106773 392 external_mini_cluster.cc:1428] Setting key 9e17a92c6bf68e91dbd99249f79b369a
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:23.110903 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.101904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42701 (local address 127.0.98.1:42433)
0927 21:07:23.102039 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:23.102082 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:23.103013 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:23.103312 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:23.103316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:23.103576 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:23.103764 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:23.103768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.104470 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.104471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:23.105268 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:23.105271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.105681 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.105682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:23.105808 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.106227 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:23.106234 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:23.107901 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:23.109511 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:23.109516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:23.109518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:23.109828 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:23.110073 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:23.110074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:23.110074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:23.110168 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:23.110449 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:23.110453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:23.110561 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 21:07:23.110639 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:07:23.110756 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260927 21:07:23.110941 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.101945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42433 (local address 127.0.98.62:42701)
0927 21:07:23.101989 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:23.101993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:23.102165 (+ 172us) server_negotiation.cc:407] Connection header received
0927 21:07:23.103151 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:23.103153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:23.103168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:23.103232 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:23.103871 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.104377 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.105359 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.105585 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.105671 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.107987 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:23.107989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:23.107989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:23.107997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:23.109430 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.109945 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.109946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.109946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.110008 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.110234 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.110235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.110235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.110385 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:23.110455 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:23.110684 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 21:07:23.110796 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:23.111464 4148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42701
I20260927 21:07:23.111562 4148 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:23.111749 4148 heartbeater.cc:511] Master 127.0.98.62:42701 requested a full tablet report, sending...
I20260927 21:07:23.112253 3957 ts_manager.cc:194] Registered new tserver with Master: d9838b64f96a48b2bb18ed2c3f844f47 (127.0.98.1:44209)
I20260927 21:07:23.112828 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42433
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:23.131378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:23.209048 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:23.209234 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:23.209255 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:23.210397 4153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:23.210434 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:23.210491 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:23.211891 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:23.212116 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:23.212327 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:23.217024 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:23.217036 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:23.217043 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:23.217253 4153 server_base.cc:1034] running on GCE node
I20260927 21:07:23.217594 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:23.217832 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:23.218995 4153 hybrid_clock.cc:648] HybridClock initialized: now 1790543243218975 us; error 31 us; skew 500 ppm
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:23.221365 4153 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:23.221792 4153 webserver.cc:460] Webserver started at http://127.0.98.2:44625/ using document root <none> and password file <none>
I20260927 21:07:23.222007 4153 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:23.222074 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:23.222225 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:23.223321 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "09cd243e1c074d89b1419fb7e7fd667d"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "5daac903ff3ab6f4d5312ffbb96998ca"
server_key_iv: "5573e85520bea09d04a2ea7dc7367208"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.223680 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "09cd243e1c074d89b1419fb7e7fd667d"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "5daac903ff3ab6f4d5312ffbb96998ca"
server_key_iv: "5573e85520bea09d04a2ea7dc7367208"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.224947 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:23.225615 4168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.225773 4153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:23.225867 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "09cd243e1c074d89b1419fb7e7fd667d"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "5daac903ff3ab6f4d5312ffbb96998ca"
server_key_iv: "5573e85520bea09d04a2ea7dc7367208"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.225946 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:23.240299 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:23.242007 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:23.242172 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:23.242406 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:23.242702 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:23.242734 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.242786 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:23.242816 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.248361 4153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35121
I20260927 21:07:23.248395 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35121 every 8 connection(s)
I20260927 21:07:23.248725 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:23.255395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4153
I20260927 21:07:23.255507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:23.255723 392 external_mini_cluster.cc:1428] Setting key 7780e329d5109cdeff1b05d19343b2e0
I20260927 21:07:23.257933 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.249436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50509 (local address 127.0.98.62:42701)
0927 21:07:23.249522 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:23.249524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:23.249692 (+ 168us) server_negotiation.cc:407] Connection header received
0927 21:07:23.250635 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:23.250637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:23.250649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:23.250702 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:23.251310 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.251833 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.252835 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.253047 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.253131 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.254982 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:23.254983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:23.254984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:23.254991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:23.256521 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.256916 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.256917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.256917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.256988 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.257251 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.257252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.257252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.257406 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:23.257484 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:23.257809 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 21:07:23.257873 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:07:23.258198 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.249373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42701 (local address 127.0.98.2:50509)
0927 21:07:23.249562 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:23.249609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:23.250525 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:23.250778 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:23.250782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:23.251022 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:23.251214 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:23.251218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.251938 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.251939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:23.252735 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:23.252738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.253139 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.253141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:23.253262 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.253654 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:23.253661 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:23.254881 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:23.256610 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:23.256616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:23.256618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:23.256827 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:23.257053 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:23.257054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:23.257055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:23.257155 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:23.257492 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:23.257498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:23.257669 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 21:07:23.257788 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:07:23.257952 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260927 21:07:23.258703 4282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42701
I20260927 21:07:23.258790 4282 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:23.258966 4282 heartbeater.cc:511] Master 127.0.98.62:42701 requested a full tablet report, sending...
I20260927 21:07:23.259325 3957 ts_manager.cc:194] Registered new tserver with Master: 09cd243e1c074d89b1419fb7e7fd667d (127.0.98.2:35121)
I20260927 21:07:23.259634 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50509
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:23.277770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:23.356350 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:23.356587 4287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:23.356623 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:23.357748 4287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:23.357815 4287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:23.357858 4287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:23.359331 4287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:23.359589 4287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:23.359807 4287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:23.364552 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:23.364563 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:23.364563 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:23.364899 4287 server_base.cc:1034] running on GCE node
I20260927 21:07:23.365118 4287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:23.365357 4287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:23.366520 4287 hybrid_clock.cc:648] HybridClock initialized: now 1790543243366497 us; error 34 us; skew 500 ppm
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:23.368947 4287 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:23.369415 4287 webserver.cc:460] Webserver started at http://127.0.98.3:35913/ using document root <none> and password file <none>
I20260927 21:07:23.369638 4287 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:23.369705 4287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:23.369863 4287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:23.370889 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e8645275de74d668abddc951ce26408"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "265337913dd6400d15a4f054efec192f"
server_key_iv: "a0d2e3d9a65b7efce25e1718b603b4d4"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.371276 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e8645275de74d668abddc951ce26408"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "265337913dd6400d15a4f054efec192f"
server_key_iv: "a0d2e3d9a65b7efce25e1718b603b4d4"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.372573 4287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:23.373241 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.373409 4287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:23.373510 4287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "8e8645275de74d668abddc951ce26408"
format_stamp: "Formatted at 2026-09-27 21:07:23 on dist-test-slave-8bgs"
server_key: "265337913dd6400d15a4f054efec192f"
server_key_iv: "a0d2e3d9a65b7efce25e1718b603b4d4"
server_key_version: "encryptionkey@0"
I20260927 21:07:23.373582 4287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:23.388972 4287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:23.390744 4287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:23.390910 4287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:23.391181 4287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:23.391561 4287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:23.391628 4287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.391671 4287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:23.391697 4287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:23.397171 4287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37687
I20260927 21:07:23.397214 4415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37687 every 8 connection(s)
I20260927 21:07:23.397519 4287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:23.401861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4287
I20260927 21:07:23.401978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:23.402175 392 external_mini_cluster.cc:1428] Setting key 0c791dbb17fc6a273f8eda7ec5c63305
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:23.406129 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.398141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45031 (local address 127.0.98.62:42701)
0927 21:07:23.398220 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:23.398222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:23.398414 (+ 192us) server_negotiation.cc:407] Connection header received
0927 21:07:23.399373 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:23.399374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:23.399384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:23.399434 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:23.400013 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.400484 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.401452 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.401625 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.401700 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.403918 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:23.403919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:23.403920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:23.403925 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:23.405053 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.405407 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.405408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.405408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.405455 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.405660 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.405661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.405661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.405751 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:23.405799 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:23.406014 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 21:07:23.406069 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:07:23.406352 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.398121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42701 (local address 127.0.98.3:45031)
0927 21:07:23.398299 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:23.398332 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:23.399283 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:23.399500 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:23.399503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:23.399745 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:23.399925 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:23.399929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.400574 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.400575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:23.401373 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:23.401375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.401696 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.401697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:23.401792 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.402354 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:23.402367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:23.403839 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:23.405122 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:23.405127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:23.405129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:23.405345 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:23.405511 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:23.405512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:23.405512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:23.405602 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:23.405805 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:23.405809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:23.405913 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 21:07:23.405990 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:07:23.406099 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260927 21:07:23.406862 4416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42701
I20260927 21:07:23.406970 4416 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:23.407194 4416 heartbeater.cc:511] Master 127.0.98.62:42701 requested a full tablet report, sending...
I20260927 21:07:23.407537 3957 ts_manager.cc:194] Registered new tserver with Master: 8e8645275de74d668abddc951ce26408 (127.0.98.3:37687)
I20260927 21:07:23.407902 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45031
I20260927 21:07:23.413731 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:23.435928 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.428400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.98.1:44209)
0927 21:07:23.428602 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:23.428607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:23.428616 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:23.428640 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:23.428642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:23.428656 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:23.428738 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:23.429294 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.430128 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.430863 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.431240 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.431316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.433365 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:23.433372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:23.433372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:23.433379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:23.434687 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.435044 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.435047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.435048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.435146 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.435358 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.435358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.435359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.435495 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:23.435647 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:23.435730 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:07:23.435810 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:23.443349 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:23.437041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.98.62:42701)
0927 21:07:23.437097 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:23.437099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:23.437145 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:07:23.437189 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:23.437190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:23.437198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:23.437244 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:23.437658 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.438105 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.438759 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:23.438934 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:23.438985 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:23.441105 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:23.441106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:23.441106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:23.441112 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:23.442261 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.442596 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.442597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.442597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.442670 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:23.442858 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:23.442858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:23.442859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:23.442990 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:23.443065 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:23.443195 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:07:23.443251 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 21:07:23.443612 3957 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39710
I20260927 21:07:23.443967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4019
I20260927 21:07:23.448722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4153
I20260927 21:07:23.452848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4287
I20260927 21:07:23.456957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 3925
2026-09-27T21:07:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1785 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[4428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[4428](info): set up 2 sockets
Sep 27 21:07:23 dist-test-slave-8bgs krb5kdc[4428](info): commencing operation
krb5kdc: starting...
W20260927 21:07:24.483011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:24 dist-test-slave-8bgs krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:24.594084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33691
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:24.672171 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:24.672366 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:24.672396 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:24.674105 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:24.674166 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:24.674196 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:24.674209 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:24.674221 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:24.675774 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:24.676021 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:24.676244 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:24.680804 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:24.680814 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:24.680814 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:24.680853 4441 server_base.cc:1034] running on GCE node
I20260927 21:07:24.681272 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:24.681522 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:24.682660 4441 hybrid_clock.cc:648] HybridClock initialized: now 1790543244682646 us; error 28 us; skew 500 ppm
Sep 27 21:07:24 dist-test-slave-8bgs krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:24.685122 4441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:24.685403 4441 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:24.685446 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:24.685530 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:24.686830 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "735e9846905e4e948f6913d70e989267"
format_stamp: "Formatted at 2026-09-27 21:07:24 on dist-test-slave-8bgs"
server_key: "0715da69e8d311881d295ba5491e5d1f"
server_key_iv: "4813fb8e846cc02f1c583a33b209f32a"
server_key_version: "encryptionkey@0"
I20260927 21:07:24.687288 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "735e9846905e4e948f6913d70e989267"
format_stamp: "Formatted at 2026-09-27 21:07:24 on dist-test-slave-8bgs"
server_key: "0715da69e8d311881d295ba5491e5d1f"
server_key_iv: "4813fb8e846cc02f1c583a33b209f32a"
server_key_version: "encryptionkey@0"
I20260927 21:07:24.688560 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:24.689201 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:24.689365 4441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:24.689464 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "735e9846905e4e948f6913d70e989267"
format_stamp: "Formatted at 2026-09-27 21:07:24 on dist-test-slave-8bgs"
server_key: "0715da69e8d311881d295ba5491e5d1f"
server_key_iv: "4813fb8e846cc02f1c583a33b209f32a"
server_key_version: "encryptionkey@0"
I20260927 21:07:24.689541 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:24.700212 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:24.702332 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:24.702479 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:24.706142 4441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33691
I20260927 21:07:24.706180 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33691 every 8 connection(s)
I20260927 21:07:24.706432 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:24.707298 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:24.708285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4441
I20260927 21:07:24.708395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:24.708566 392 external_mini_cluster.cc:1428] Setting key 2d3ff043c2f93ba23703718f63347735
I20260927 21:07:24.709786 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: Bootstrap starting.
I20260927 21:07:24.710515 4508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:24.710815 4508 log.cc:824] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:24.711500 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: No bootstrap required, opened a new log
I20260927 21:07:24.713706 4508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735e9846905e4e948f6913d70e989267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33691 } }
I20260927 21:07:24.713879 4508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:24.713914 4508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735e9846905e4e948f6913d70e989267, State: Initialized, Role: FOLLOWER
I20260927 21:07:24.714085 4508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735e9846905e4e948f6913d70e989267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33691 } }
I20260927 21:07:24.714186 4508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:24.714222 4508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:24.714277 4508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:24.714884 4508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735e9846905e4e948f6913d70e989267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33691 } }
I20260927 21:07:24.714965 4508 leader_election.cc:302] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 735e9846905e4e948f6913d70e989267; no voters:
Sep 27 21:07:24 dist-test-slave-8bgs krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:24.715173 4508 leader_election.cc:288] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:24.715206 4513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:24.715386 4513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [term 1 LEADER]: Becoming Leader. State: Replica: 735e9846905e4e948f6913d70e989267, State: Running, Role: LEADER
I20260927 21:07:24.715440 4508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:24.715539 4513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735e9846905e4e948f6913d70e989267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33691 } }
I20260927 21:07:24.715898 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "735e9846905e4e948f6913d70e989267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735e9846905e4e948f6913d70e989267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33691 } } }
I20260927 21:07:24.715979 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:24.716925 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 735e9846905e4e948f6913d70e989267. Latest consensus state: current_term: 1 leader_uuid: "735e9846905e4e948f6913d70e989267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735e9846905e4e948f6913d70e989267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33691 } } }
I20260927 21:07:24.717015 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267 [sys.catalog]: This master's current role is: LEADER
W20260927 21:07:24.717165 4529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:07:24.717217 4529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:07:24.719048 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:24.709047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.0.98.62:33691)
0927 21:07:24.709217 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:24.709224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:24.709247 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:07:24.709632 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:24.709638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:24.709680 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:24.709788 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:24.710604 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:24.711614 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:24.712602 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:24.712972 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:24.713042 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:24.715724 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:24.715728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:24.715728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:24.715741 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:24.717653 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:24.718010 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:24.718013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:24.718013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:24.718108 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:24.718302 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:24.718303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:24.718303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:24.718523 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:24.718677 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:24.718766 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 21:07:24.718889 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260927 21:07:24.719142 4521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:24.719327 4521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:24.720765 4521 catalog_manager.cc:1347] Generated new cluster ID: c62927b6bf55483eb141209be7aff715
I20260927 21:07:24.720817 4521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:24.726364 4521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:24.726846 4521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:24.732522 4521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: Generated new TSK 0
I20260927 21:07:24.732707 4521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:24.765259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33691
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:24.859851 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:24.860067 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:24.860106 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:24.861245 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:24.861307 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:24.861351 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:24.862802 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:24.863049 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:24.863279 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:24.867991 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:24.867974 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:24.867975 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:24.868253 4534 server_base.cc:1034] running on GCE node
I20260927 21:07:24.868461 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:24.868727 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:24.869880 4534 hybrid_clock.cc:648] HybridClock initialized: now 1790543244869875 us; error 38 us; skew 500 ppm
Sep 27 21:07:24 dist-test-slave-8bgs krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:24.872315 4534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:24.872673 4534 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:24.872741 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:24.872876 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:24.874027 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ad97609049f49e8af3d5efd59f07ef1"
format_stamp: "Formatted at 2026-09-27 21:07:24 on dist-test-slave-8bgs"
server_key: "f33633c08f0478313609a414ee5b2d61"
server_key_iv: "7f29533b14d9628a720e2e909df057c4"
server_key_version: "encryptionkey@0"
I20260927 21:07:24.874379 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ad97609049f49e8af3d5efd59f07ef1"
format_stamp: "Formatted at 2026-09-27 21:07:24 on dist-test-slave-8bgs"
server_key: "f33633c08f0478313609a414ee5b2d61"
server_key_iv: "7f29533b14d9628a720e2e909df057c4"
server_key_version: "encryptionkey@0"
I20260927 21:07:24.875679 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:24.876348 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:24.876499 4534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:24.876592 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "4ad97609049f49e8af3d5efd59f07ef1"
format_stamp: "Formatted at 2026-09-27 21:07:24 on dist-test-slave-8bgs"
server_key: "f33633c08f0478313609a414ee5b2d61"
server_key_iv: "7f29533b14d9628a720e2e909df057c4"
server_key_version: "encryptionkey@0"
I20260927 21:07:24.876677 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:24.898836 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:24.900630 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:24.900791 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:24.901016 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:24.901324 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:24.901376 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:24.901417 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:24.901444 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:24.907145 4534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41219
I20260927 21:07:24.907195 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41219 every 8 connection(s)
I20260927 21:07:24.907438 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:24.910878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4534
I20260927 21:07:24.911000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:24.911235 392 external_mini_cluster.cc:1428] Setting key d91c19eaa52e521b1c238e3ec471074b
Sep 27 21:07:24 dist-test-slave-8bgs krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:24.916814 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:24.908136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52465 (local address 127.0.98.62:33691)
0927 21:07:24.908204 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:24.908206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:24.908419 (+ 213us) server_negotiation.cc:407] Connection header received
0927 21:07:24.908854 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:24.908855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:24.908865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:24.908919 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:24.909807 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:24.910300 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:24.911398 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:24.911695 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:24.911772 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:24.913833 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:24.913835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:24.913835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:24.913843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:24.915268 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:24.915773 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:24.915774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:24.915774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:24.915829 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:24.916110 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:24.916111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:24.916111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:24.916214 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:24.916264 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:24.916661 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 21:07:24.916750 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:07:24.917070 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:24.908129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33691 (local address 127.0.98.1:52465)
0927 21:07:24.908295 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:24.908329 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:24.908770 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:24.908993 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:24.908996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:24.909255 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:24.909727 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:24.909732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:24.910374 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:24.910374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:24.911297 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:24.911299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:24.911777 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:24.911778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:24.911924 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:24.912439 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:24.912452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:24.913739 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:24.915356 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:24.915363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:24.915366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:24.915696 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:24.915897 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:24.915898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:24.915898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:24.916046 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:24.916278 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:24.916284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:24.916466 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 21:07:24.916593 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:07:24.916788 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260927 21:07:24.917692 4662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33691
I20260927 21:07:24.917800 4662 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:24.917960 4662 heartbeater.cc:511] Master 127.0.98.62:33691 requested a full tablet report, sending...
I20260927 21:07:24.918514 4472 ts_manager.cc:194] Registered new tserver with Master: 4ad97609049f49e8af3d5efd59f07ef1 (127.0.98.1:41219)
I20260927 21:07:24.919519 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52465
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:24.935534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33691
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:25.016392 4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:25.016578 4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:25.016601 4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:25.017768 4667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:25.017808 4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:25.017872 4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:25.019451 4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:25.019703 4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:25.019935 4667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:25.024771 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.024767 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.024747 4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.024891 4667 server_base.cc:1034] running on GCE node
I20260927 21:07:25.025170 4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:25.025432 4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:25.026587 4667 hybrid_clock.cc:648] HybridClock initialized: now 1790543245026581 us; error 34 us; skew 500 ppm
Sep 27 21:07:25 dist-test-slave-8bgs krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:25.029024 4667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:25.029404 4667 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:25.029474 4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:25.029613 4667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:25.030655 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b76aa09497d41cb956d5c3bfb8aac87"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "d39e183cf3ffb14386b1d2dac23b24c9"
server_key_iv: "79d11a7f7bf361770aeb70d59d0f6413"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.031028 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b76aa09497d41cb956d5c3bfb8aac87"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "d39e183cf3ffb14386b1d2dac23b24c9"
server_key_iv: "79d11a7f7bf361770aeb70d59d0f6413"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.032321 4667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:25.033028 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.033205 4667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:25.033303 4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "5b76aa09497d41cb956d5c3bfb8aac87"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "d39e183cf3ffb14386b1d2dac23b24c9"
server_key_iv: "79d11a7f7bf361770aeb70d59d0f6413"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.033389 4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:25.043519 4667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:25.045193 4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:25.045347 4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:25.045557 4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:25.045881 4667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:25.045934 4667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.045967 4667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:25.045996 4667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.051788 4667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46569
I20260927 21:07:25.051836 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46569 every 8 connection(s)
I20260927 21:07:25.052075 4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:07:25 dist-test-slave-8bgs krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:25.059975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4667
I20260927 21:07:25.060081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:25.060250 392 external_mini_cluster.cc:1428] Setting key f9b43216d9d59b69ac9bf8f0e8110ee3
I20260927 21:07:25.060941 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.052726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36059 (local address 127.0.98.62:33691)
0927 21:07:25.052795 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.052797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.053015 (+ 218us) server_negotiation.cc:407] Connection header received
0927 21:07:25.053520 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.053522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.053533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.053584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.054452 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.054932 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.055946 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.056151 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.056217 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.058199 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.058200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.058200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:25.058208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.059566 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:25.059980 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:25.059981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:25.059982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:25.060062 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:25.060297 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:25.060298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:25.060298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:25.060447 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.060509 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.060769 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.060853 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:07:25.061122 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.052713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33691 (local address 127.0.98.2:36059)
0927 21:07:25.052903 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.052935 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.053417 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.053656 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.053659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.053896 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.054353 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.054358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.055029 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.055030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.055845 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.055847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.056234 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.056234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.056350 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.056815 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:25.056822 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.058088 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.059660 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:25.059666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:25.059669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:25.059892 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:25.060128 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:25.060129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:25.060129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:25.060228 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:25.060501 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.060507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.060644 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.060749 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.060894 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260927 21:07:25.061537 4795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33691
I20260927 21:07:25.061618 4795 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:25.061776 4795 heartbeater.cc:511] Master 127.0.98.62:33691 requested a full tablet report, sending...
I20260927 21:07:25.062115 4472 ts_manager.cc:194] Registered new tserver with Master: 5b76aa09497d41cb956d5c3bfb8aac87 (127.0.98.2:46569)
I20260927 21:07:25.062526 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36059
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:25.097262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33691
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:25.175518 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:25.175752 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:25.175789 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:25.176957 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:25.177021 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:25.177069 4800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:25.178527 4800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:25.178779 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:25.178995 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:25.183874 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.183856 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.183959 4800 server_base.cc:1034] running on GCE node
W20260927 21:07:25.183897 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.184247 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:25.184491 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:25.185631 4800 hybrid_clock.cc:648] HybridClock initialized: now 1790543245185620 us; error 29 us; skew 500 ppm
Sep 27 21:07:25 dist-test-slave-8bgs krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:25.188050 4800 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:25.188328 4800 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:25.188369 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:25.188453 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:25.189401 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "81b52d43fa6c4b5fab4eee6de083babf"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "da0ccdb8a104557b1fbb1544e232fed0"
server_key_iv: "d6d95702c42b25dca7bb3dc13ff5ad5d"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.189702 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81b52d43fa6c4b5fab4eee6de083babf"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "da0ccdb8a104557b1fbb1544e232fed0"
server_key_iv: "d6d95702c42b25dca7bb3dc13ff5ad5d"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.190871 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:25.191610 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.191787 4800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.191884 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "81b52d43fa6c4b5fab4eee6de083babf"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "da0ccdb8a104557b1fbb1544e232fed0"
server_key_iv: "d6d95702c42b25dca7bb3dc13ff5ad5d"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.191960 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:25.207502 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:25.209143 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:25.209299 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:25.209512 4800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:25.209810 4800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:25.209861 4800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.209895 4800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:25.209921 4800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.215461 4800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34417
I20260927 21:07:25.215502 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34417 every 8 connection(s)
I20260927 21:07:25.215801 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:25 dist-test-slave-8bgs krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:25.221835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4800
I20260927 21:07:25.221935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:25.222178 392 external_mini_cluster.cc:1428] Setting key f026e7928b2e7f5135913f6ec818d4fa
I20260927 21:07:25.224753 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.216502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45555 (local address 127.0.98.62:33691)
0927 21:07:25.216585 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.216587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.216794 (+ 207us) server_negotiation.cc:407] Connection header received
0927 21:07:25.217256 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.217257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.217266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.217323 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.218146 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.218621 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.219605 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.219791 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.219847 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.221827 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.221829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.221830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:25.221837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.223389 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:25.223813 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:25.223814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:25.223814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:25.223871 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:25.224123 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:25.224124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:25.224125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:25.224249 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.224325 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.224587 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.224659 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:07:25.224949 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.216507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33691 (local address 127.0.98.3:45555)
0927 21:07:25.216678 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.216718 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.217188 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.217372 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.217375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.217610 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.218072 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.218076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.218694 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.218695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.219533 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.219536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.219862 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.219862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.219954 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.220449 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:25.220456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.221727 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.223501 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:25.223509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:25.223511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:25.223735 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:25.223943 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:25.223944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:25.223945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:25.224054 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:25.224317 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.224324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.224460 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.224567 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.224726 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260927 21:07:25.225397 4928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33691
I20260927 21:07:25.225471 4928 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:25.225634 4928 heartbeater.cc:511] Master 127.0.98.62:33691 requested a full tablet report, sending...
I20260927 21:07:25.225994 4472 ts_manager.cc:194] Registered new tserver with Master: 81b52d43fa6c4b5fab4eee6de083babf (127.0.98.3:34417)
I20260927 21:07:25.226315 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45555
I20260927 21:07:25.233740 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:07:25.241113 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.234984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.0.98.62:33691)
0927 21:07:25.235042 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.235044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.235148 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:07:25.235230 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.235231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.235239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.235287 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.235784 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.236392 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.237437 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.237639 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.237728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.238837 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.238838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.238838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:25.238845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.239961 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:25.240321 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:25.240323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:25.240323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:25.240380 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:25.240577 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:25.240578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:25.240579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:25.240721 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.240826 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.240937 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.241012 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:07:25.242715 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:25.244356 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:25.250515 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.246548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41219 (local address 127.0.0.1:59246)
0927 21:07:25.246699 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.246741 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.246808 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.247276 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.247279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.247282 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:25.247367 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.247369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.248271 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.248272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.249421 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.249424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.250231 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.250232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.250308 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.250322 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.250394 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.250441 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":43,"threads_started":1}
I20260927 21:07:25.250555 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.246730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.0.98.1:41219)
0927 21:07:25.247077 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.247103 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.247111 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:25.247146 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.247148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.247164 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.247446 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:25.247527 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.248152 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.249499 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.250145 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.250220 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.250277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.250419 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.250463 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":71,"threads_started":1}
I20260927 21:07:25.250850 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.246726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46569 (local address 127.0.0.1:55866)
0927 21:07:25.246922 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.246930 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.246970 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.247335 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.247336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.247338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:25.247404 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.247406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.248659 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.248660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.249664 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.249666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.250608 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.250610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.250682 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.250686 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.250746 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.250787 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260927 21:07:25.250885 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.246928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.0.98.2:46569)
0927 21:07:25.247100 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.247105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.247115 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:25.247160 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.247163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.247178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.247253 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:25.247485 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.248557 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.249736 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.250459 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.250524 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.250573 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.250772 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.250809 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260927 21:07:25.251713 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.246548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.0.1:45538)
0927 21:07:25.246886 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.246897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.246957 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.247292 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.247293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.247294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:25.247371 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.247373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.249731 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.249732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.250501 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.250503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.251450 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.251451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.251514 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.251518 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.251600 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.251640 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":31,"thread_start_us":87,"threads_started":1}
I20260927 21:07:25.251730 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.246658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.0.98.3:34417)
0927 21:07:25.246843 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.246848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.247025 (+ 177us) server_negotiation.cc:407] Connection header received
0927 21:07:25.247072 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.247076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.247133 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.247210 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:25.247429 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.248552 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.250589 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.251350 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.251446 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.251501 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.251616 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.251656 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260927 21:07:25.251684 4596 tablet_service.cc:1467] Processing CreateTablet for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 (DEFAULT_TABLE table=test-table [id=d88b76fc739447cf85180136687f9bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:25.251684 4729 tablet_service.cc:1467] Processing CreateTablet for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 (DEFAULT_TABLE table=test-table [id=d88b76fc739447cf85180136687f9bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:25.252022 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.252022 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.252606 4862 tablet_service.cc:1467] Processing CreateTablet for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 (DEFAULT_TABLE table=test-table [id=d88b76fc739447cf85180136687f9bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:25.252899 4862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.255343 4946 tablet_bootstrap.cc:492] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: Bootstrap starting.
I20260927 21:07:25.255934 4946 tablet_bootstrap.cc:654] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.256440 4946 log.cc:824] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.256747 4947 tablet_bootstrap.cc:492] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Bootstrap starting.
I20260927 21:07:25.257086 4946 tablet_bootstrap.cc:492] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: No bootstrap required, opened a new log
I20260927 21:07:25.257141 4946 ts_tablet_manager.cc:1397] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:25.257318 4947 tablet_bootstrap.cc:654] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.257828 4947 log.cc:824] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.258109 4945 tablet_bootstrap.cc:492] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Bootstrap starting.
I20260927 21:07:25.258762 4945 tablet_bootstrap.cc:654] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.258881 4946 raft_consensus.cc:348] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.259028 4946 raft_consensus.cc:374] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.259051 4946 raft_consensus.cc:729] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad97609049f49e8af3d5efd59f07ef1, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.259217 4946 consensus_queue.cc:260] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.259352 4945 log.cc:824] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.259493 4947 tablet_bootstrap.cc:492] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: No bootstrap required, opened a new log
I20260927 21:07:25.259495 4946 ts_tablet_manager.cc:1428] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260927 21:07:25.259570 4947 ts_tablet_manager.cc:1397] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:07:25.259642 4662 heartbeater.cc:503] Master 127.0.98.62:33691 was elected leader, sending a full tablet report...
I20260927 21:07:25.260075 4945 tablet_bootstrap.cc:492] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: No bootstrap required, opened a new log
I20260927 21:07:25.260154 4945 ts_tablet_manager.cc:1397] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:25.261355 4947 raft_consensus.cc:348] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.261473 4947 raft_consensus.cc:374] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.261493 4947 raft_consensus.cc:729] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b52d43fa6c4b5fab4eee6de083babf, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.261602 4947 consensus_queue.cc:260] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.261880 4947 ts_tablet_manager.cc:1428] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:07:25.261801 4945 raft_consensus.cc:348] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.261885 4928 heartbeater.cc:503] Master 127.0.98.62:33691 was elected leader, sending a full tablet report...
I20260927 21:07:25.261952 4945 raft_consensus.cc:374] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.261996 4945 raft_consensus.cc:729] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b76aa09497d41cb956d5c3bfb8aac87, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.262100 4945 consensus_queue.cc:260] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.262372 4945 ts_tablet_manager.cc:1428] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:25.262375 4795 heartbeater.cc:503] Master 127.0.98.62:33691 was elected leader, sending a full tablet report...
W20260927 21:07:25.302587 4796 tablet.cc:2369] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:25.362607 4953 raft_consensus.cc:482] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:25.362788 4953 raft_consensus.cc:504] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.363198 4953 leader_election.cc:288] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ad97609049f49e8af3d5efd59f07ef1 (127.0.98.1:41219), 81b52d43fa6c4b5fab4eee6de083babf (127.0.98.3:34417)
I20260927 21:07:25.366134 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.363423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39715 (local address 127.0.98.1:41219)
0927 21:07:25.363496 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.363498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.363510 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:07:25.363542 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.363543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.363555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.363638 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:25.363797 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.364489 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.365420 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.365784 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.365876 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.365886 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.366040 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.366076 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:07:25.366133 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.363359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41219 (local address 127.0.98.2:39715)
0927 21:07:25.363413 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.363426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.363485 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.363643 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.363644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.363645 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:25.363723 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.363724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.364577 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.364578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.365314 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.365316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.365877 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.365878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.365958 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.365960 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.366021 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.366062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:07:25.366279 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.363367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34417 (local address 127.0.98.2:47921)
0927 21:07:25.363585 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.363597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.363670 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.363825 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.363826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.363828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:25.363903 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.363904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.364737 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.364738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.365464 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:25.365466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.366012 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.366013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:25.366091 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.366099 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.366162 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.366198 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":10,"thread_start_us":137,"threads_started":1}
I20260927 21:07:25.366271 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.363491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47921 (local address 127.0.98.3:34417)
0927 21:07:25.363531 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.363533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.363660 (+ 127us) server_negotiation.cc:407] Connection header received
0927 21:07:25.363733 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.363734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.363752 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.363808 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:25.363969 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.364652 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.365554 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.365928 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.366011 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.366020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.366179 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.366211 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:25.366662 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad38ecddc5b40a8b1dcb64e51b9cfa3" candidate_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b52d43fa6c4b5fab4eee6de083babf" is_pre_election: true
I20260927 21:07:25.366806 4882 raft_consensus.cc:2454] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b76aa09497d41cb956d5c3bfb8aac87 in term 0.
I20260927 21:07:25.366812 4616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad38ecddc5b40a8b1dcb64e51b9cfa3" candidate_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad97609049f49e8af3d5efd59f07ef1" is_pre_election: true
I20260927 21:07:25.366968 4616 raft_consensus.cc:2454] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b76aa09497d41cb956d5c3bfb8aac87 in term 0.
I20260927 21:07:25.367004 4683 leader_election.cc:302] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b76aa09497d41cb956d5c3bfb8aac87, 81b52d43fa6c4b5fab4eee6de083babf; no voters:
I20260927 21:07:25.367115 4953 raft_consensus.cc:2781] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:25.367169 4953 raft_consensus.cc:482] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:25.367201 4953 raft_consensus.cc:3037] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:25.367844 4953 raft_consensus.cc:504] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.367985 4953 leader_election.cc:288] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [CANDIDATE]: Term 1 election: Requested vote from peers 4ad97609049f49e8af3d5efd59f07ef1 (127.0.98.1:41219), 81b52d43fa6c4b5fab4eee6de083babf (127.0.98.3:34417)
I20260927 21:07:25.368113 4616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad38ecddc5b40a8b1dcb64e51b9cfa3" candidate_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad97609049f49e8af3d5efd59f07ef1"
I20260927 21:07:25.368135 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad38ecddc5b40a8b1dcb64e51b9cfa3" candidate_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b52d43fa6c4b5fab4eee6de083babf"
I20260927 21:07:25.368204 4616 raft_consensus.cc:3037] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:25.368214 4882 raft_consensus.cc:3037] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:25.368887 4616 raft_consensus.cc:2454] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b76aa09497d41cb956d5c3bfb8aac87 in term 1.
I20260927 21:07:25.368911 4882 raft_consensus.cc:2454] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b76aa09497d41cb956d5c3bfb8aac87 in term 1.
I20260927 21:07:25.369081 4682 leader_election.cc:302] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad97609049f49e8af3d5efd59f07ef1, 5b76aa09497d41cb956d5c3bfb8aac87; no voters:
I20260927 21:07:25.369184 4953 raft_consensus.cc:2781] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:25.369318 4953 raft_consensus.cc:686] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 1 LEADER]: Becoming Leader. State: Replica: 5b76aa09497d41cb956d5c3bfb8aac87, State: Running, Role: LEADER
I20260927 21:07:25.369423 4953 consensus_queue.cc:237] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } }
I20260927 21:07:25.370210 4472 catalog_manager.cc:5495] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b76aa09497d41cb956d5c3bfb8aac87 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b76aa09497d41cb956d5c3bfb8aac87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46569 } health_report { overall_health: HEALTHY } } }
I20260927 21:07:25.397192 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.394992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.0.98.2:46569)
0927 21:07:25.395034 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.395036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.395045 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:25.395147 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.395147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.395189 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.395271 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:25.395374 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.395893 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.396581 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:25.396758 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:25.396806 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:25.396922 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:25.397016 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:25.397071 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.397100 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.397115 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:25.398921 4616 raft_consensus.cc:1261] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1 [term 1 FOLLOWER]: Refusing update from remote peer 5b76aa09497d41cb956d5c3bfb8aac87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:25.398923 4882 raft_consensus.cc:1261] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 1 FOLLOWER]: Refusing update from remote peer 5b76aa09497d41cb956d5c3bfb8aac87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:25.399255 4953 consensus_queue.cc:1035] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad97609049f49e8af3d5efd59f07ef1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:25.399381 4955 consensus_queue.cc:1035] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b52d43fa6c4b5fab4eee6de083babf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:25.403223 4959 mvcc.cc:205] Tried to move back new op lower bound from 7334065133151293440 to 7334065133033664512. Current Snapshot: MvccSnapshot[applied={T|T < 7334065133151293440}]
I20260927 21:07:25.404047 4471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:25.404139 4471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:25.404816 4471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 735e9846905e4e948f6913d70e989267: Sending DeleteTablet for 3 replicas of tablet aad38ecddc5b40a8b1dcb64e51b9cfa3
I20260927 21:07:25.405146 4596 tablet_service.cc:1514] Processing DeleteTablet for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59246
I20260927 21:07:25.405159 4862 tablet_service.cc:1514] Processing DeleteTablet for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45538
I20260927 21:07:25.405198 4729 tablet_service.cc:1514] Processing DeleteTablet for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55866
I20260927 21:07:25.405388 4968 tablet_replica.cc:330] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 4ad97609049f49e8af3d5efd59f07ef1: stopping tablet replica
I20260927 21:07:25.405472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4534
I20260927 21:07:25.406574 4970 tablet_replica.cc:330] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: stopping tablet replica
I20260927 21:07:25.406692 4970 raft_consensus.cc:2229] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:25.406801 4970 raft_consensus.cc:2258] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:25.407281 4970 ts_tablet_manager.cc:1905] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:25.408743 4970 ts_tablet_manager.cc:1918] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:25.408833 4970 log.cc:1196] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/aad38ecddc5b40a8b1dcb64e51b9cfa3
I20260927 21:07:25.409076 4970 ts_tablet_manager.cc:1939] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 5b76aa09497d41cb956d5c3bfb8aac87: Deleting consensus metadata
I20260927 21:07:25.409435 4457 catalog_manager.cc:4841] TS 5b76aa09497d41cb956d5c3bfb8aac87 (127.0.98.2:46569): tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 (table test-table [id=d88b76fc739447cf85180136687f9bce]) successfully deleted
I20260927 21:07:25.411197 4969 tablet_replica.cc:330] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: stopping tablet replica
I20260927 21:07:25.411300 4969 raft_consensus.cc:2229] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:25.411397 4969 raft_consensus.cc:2258] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:25.412058 4969 ts_tablet_manager.cc:1905] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:07:25.412322 4456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41219 (error 108)
I20260927 21:07:25.412572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4667
I20260927 21:07:25.413618 4969 ts_tablet_manager.cc:1918] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:25.413676 4969 log.cc:1196] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/aad38ecddc5b40a8b1dcb64e51b9cfa3
I20260927 21:07:25.413786 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.412803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41219 (local address 127.0.0.1:59262)
0927 21:07:25.413694 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.413743 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":856}
I20260927 21:07:25.413939 4969 ts_tablet_manager.cc:1939] T aad38ecddc5b40a8b1dcb64e51b9cfa3 P 81b52d43fa6c4b5fab4eee6de083babf: Deleting consensus metadata
W20260927 21:07:25.413908 4456 catalog_manager.cc:4584] TS 4ad97609049f49e8af3d5efd59f07ef1 (127.0.98.1:41219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aad38ecddc5b40a8b1dcb64e51b9cfa3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41219: connect: Connection refused (error 111)
I20260927 21:07:25.414283 4457 catalog_manager.cc:4841] TS 81b52d43fa6c4b5fab4eee6de083babf (127.0.98.3:34417): tablet aad38ecddc5b40a8b1dcb64e51b9cfa3 (table test-table [id=d88b76fc739447cf85180136687f9bce]) successfully deleted
I20260927 21:07:25.418738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4800
I20260927 21:07:25.423252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4441
2026-09-27T21:07:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1966 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T21:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:25Z Disabled control of system clock
I20260927 21:07:25.449755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36335
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:07:25.530225 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:25.530413 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:25.530433 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:25.531584 4975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:25.531623 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:25.531636 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:25.531649 4975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:25.531661 4975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:25.532958 4975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:25.533191 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:25.533419 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:25.538061 4975 server_base.cc:1034] running on GCE node
W20260927 21:07:25.537998 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.538020 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.538020 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.538466 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:25.538744 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:25.539915 4975 hybrid_clock.cc:648] HybridClock initialized: now 1790543245539901 us; error 26 us; skew 500 ppm
I20260927 21:07:25.541059 4975 webserver.cc:460] Webserver started at http://127.0.98.62:41807/ using document root <none> and password file <none>
I20260927 21:07:25.541291 4975 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:25.541359 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:25.541512 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:25.542606 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "17210538af314cb3bef013431b69c142"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "f2f08f2ce5da8a904623cddb35c6ff6d"
server_key_iv: "246c687edde446d970285ff46207b040"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.542958 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "17210538af314cb3bef013431b69c142"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "f2f08f2ce5da8a904623cddb35c6ff6d"
server_key_iv: "246c687edde446d970285ff46207b040"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.544348 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:25.545018 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.545197 4975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:25.545344 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "17210538af314cb3bef013431b69c142"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "f2f08f2ce5da8a904623cddb35c6ff6d"
server_key_iv: "246c687edde446d970285ff46207b040"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.545435 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:25.548647 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:25.548897 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:25.549033 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:25.552616 4975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36335
I20260927 21:07:25.552673 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36335 every 8 connection(s)
I20260927 21:07:25.552978 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:25.553467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 4975
I20260927 21:07:25.553568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:25.553673 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.553774 392 external_mini_cluster.cc:1428] Setting key d8daa506cff0a0ba6c09e7f11fecd547
I20260927 21:07:25.556497 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.554294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.0.98.62:36335)
0927 21:07:25.554468 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.554475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.554498 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:07:25.555265 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.555272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.555312 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.555422 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.556066 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.556069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.556069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.556072 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.556109 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.556119 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.556178 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.556257 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.556326 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260927 21:07:25.557251 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142: Bootstrap starting.
I20260927 21:07:25.557783 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.558037 5042 log.cc:824] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.558772 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142: No bootstrap required, opened a new log
I20260927 21:07:25.560082 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17210538af314cb3bef013431b69c142" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260927 21:07:25.560226 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.560289 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17210538af314cb3bef013431b69c142, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.560432 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17210538af314cb3bef013431b69c142" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260927 21:07:25.560518 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:25.560560 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:25.560607 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:25.561082 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17210538af314cb3bef013431b69c142" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260927 21:07:25.561194 5042 leader_election.cc:302] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17210538af314cb3bef013431b69c142; no voters:
I20260927 21:07:25.561368 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:25.561573 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:25.561643 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:25.561722 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [term 1 LEADER]: Becoming Leader. State: Replica: 17210538af314cb3bef013431b69c142, State: Running, Role: LEADER
I20260927 21:07:25.561825 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17210538af314cb3bef013431b69c142" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } }
I20260927 21:07:25.562166 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17210538af314cb3bef013431b69c142. Latest consensus state: current_term: 1 leader_uuid: "17210538af314cb3bef013431b69c142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17210538af314cb3bef013431b69c142" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } } }
I20260927 21:07:25.562325 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:25.562178 5048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17210538af314cb3bef013431b69c142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17210538af314cb3bef013431b69c142" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36335 } } }
I20260927 21:07:25.562533 5048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:25.563216 5054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:25.563818 5054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:25.565141 5054 catalog_manager.cc:1347] Generated new cluster ID: d15dbc4bc1204cc1afa11b0b6c9e8b2c
I20260927 21:07:25.565209 5054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:25.577277 5054 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:25.577453 5054 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:25.585790 5054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142: Generated new TSK 0
I20260927 21:07:25.585976 5054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:25.597345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:07:25.687270 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:25.687490 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:25.687542 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:25.688674 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:25.688738 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:25.688783 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:25.690115 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:25.690392 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:25.690672 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:25.696223 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.696305 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.696482 5066 server_base.cc:1034] running on GCE node
W20260927 21:07:25.696224 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.696776 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:25.697034 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:25.698199 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790543245698177 us; error 37 us; skew 500 ppm
I20260927 21:07:25.699383 5066 webserver.cc:460] Webserver started at http://127.0.98.1:40699/ using document root <none> and password file <none>
I20260927 21:07:25.699625 5066 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:25.699703 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:25.699849 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:25.700862 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "a50d2ae251874f7c91b52b58fc831300"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "9e64b58ae3c7c6438d7aa8f98aa867e1"
server_key_iv: "f145df6e66dd58449eb8837df1f8c0ff"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.701226 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a50d2ae251874f7c91b52b58fc831300"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "9e64b58ae3c7c6438d7aa8f98aa867e1"
server_key_iv: "f145df6e66dd58449eb8837df1f8c0ff"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.702569 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:25.703289 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.703464 5066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:25.703560 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "a50d2ae251874f7c91b52b58fc831300"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "9e64b58ae3c7c6438d7aa8f98aa867e1"
server_key_iv: "f145df6e66dd58449eb8837df1f8c0ff"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.703635 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:25.706825 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:25.707077 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:25.707247 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:25.707453 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:25.707767 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:25.707819 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.707857 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:25.707885 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.713404 5066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38065
I20260927 21:07:25.713465 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38065 every 8 connection(s)
I20260927 21:07:25.713774 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:25.716595 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.714417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36649 (local address 127.0.98.62:36335)
0927 21:07:25.714461 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.714463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.714648 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:07:25.715117 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.715118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.715124 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.715192 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.716347 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.716349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.716349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.716354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.716368 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.716373 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.716428 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.716519 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.716533 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:25.716818 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.714384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.1:36649)
0927 21:07:25.714535 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.714563 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.714998 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.715256 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.715260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.715269 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.716206 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:25.716206 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.716230 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:25.716255 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.716428 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.716433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.716436 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.716528 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.716583 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260927 21:07:25.717288 5194 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260927 21:07:25.717367 5194 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:25.717465 5194 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260927 21:07:25.717888 5006 ts_manager.cc:194] Registered new tserver with Master: a50d2ae251874f7c91b52b58fc831300 (127.0.98.1:38065)
I20260927 21:07:25.722335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5066
I20260927 21:07:25.722443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:25.722656 392 external_mini_cluster.cc:1428] Setting key b44e9fa0c9edec69a75082d3a0824dcb
I20260927 21:07:25.723531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:07:25.801908 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:25.802120 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:25.802153 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:25.803300 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:25.803354 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:25.803395 5197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:25.804754 5197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:25.804993 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:25.805346 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:25.810550 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.810557 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.810550 5203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.810824 5197 server_base.cc:1034] running on GCE node
I20260927 21:07:25.811053 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:25.811327 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:25.812466 5197 hybrid_clock.cc:648] HybridClock initialized: now 1790543245812448 us; error 36 us; skew 500 ppm
I20260927 21:07:25.813622 5197 webserver.cc:460] Webserver started at http://127.0.98.2:40635/ using document root <none> and password file <none>
I20260927 21:07:25.813871 5197 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:25.813916 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:25.814072 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:25.815129 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "2d1be1aabcfd46409259955bb773544f"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "acdb9226e4d5e3c3df831b51ffdd8311"
server_key_iv: "b4d649f7b5c7d546335ba562080b6bd9"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.815505 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2d1be1aabcfd46409259955bb773544f"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "acdb9226e4d5e3c3df831b51ffdd8311"
server_key_iv: "b4d649f7b5c7d546335ba562080b6bd9"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.816757 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:25.817444 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.817597 5197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:25.817685 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "2d1be1aabcfd46409259955bb773544f"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "acdb9226e4d5e3c3df831b51ffdd8311"
server_key_iv: "b4d649f7b5c7d546335ba562080b6bd9"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.817765 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:25.821022 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:25.821280 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:25.821422 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:25.821626 5197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:25.821918 5197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:25.821973 5197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.822013 5197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:25.822041 5197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.827455 5197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40509
I20260927 21:07:25.827526 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40509 every 8 connection(s)
I20260927 21:07:25.827847 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:25.830461 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.828527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37717 (local address 127.0.98.62:36335)
0927 21:07:25.828573 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.828576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.828814 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:07:25.829235 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.829236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.829241 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.829304 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.830291 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.830292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.830292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.830295 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.830307 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.830310 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.830354 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.830399 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.830410 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:25.830691 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.828540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.2:37717)
0927 21:07:25.828696 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.828729 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.829149 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.829353 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.829356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.829364 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.830162 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:25.830162 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.830193 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:25.830218 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.830353 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.830355 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.830357 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.830405 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.830448 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":84,"threads_started":1}
I20260927 21:07:25.831143 5325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260927 21:07:25.831246 5325 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:25.831391 5325 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260927 21:07:25.831766 5006 ts_manager.cc:194] Registered new tserver with Master: 2d1be1aabcfd46409259955bb773544f (127.0.98.2:40509)
I20260927 21:07:25.837548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5197
I20260927 21:07:25.837661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:25.837872 392 external_mini_cluster.cc:1428] Setting key 86f1b80cceffc9e9f5a9317bd5f7a93b
I20260927 21:07:25.838726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:07:25.917636 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:25.917834 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:25.917855 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:25.919003 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:25.919041 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:25.919066 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:25.920478 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:25.920711 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:25.920979 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:25.925899 5333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:25.925904 5336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.926020 5328 server_base.cc:1034] running on GCE node
W20260927 21:07:25.926110 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:25.926304 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:25.926518 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:25.927654 5328 hybrid_clock.cc:648] HybridClock initialized: now 1790543245927632 us; error 37 us; skew 500 ppm
I20260927 21:07:25.928821 5328 webserver.cc:460] Webserver started at http://127.0.98.3:39421/ using document root <none> and password file <none>
I20260927 21:07:25.929008 5328 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:25.929049 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:25.929136 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:25.930128 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "509d7ce982d945c6b5311a3188e1fa7c"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "498652c2d7f4c6a868c52fb33d5d8485"
server_key_iv: "713523c911e10f00608cfcad66d62580"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.930433 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "509d7ce982d945c6b5311a3188e1fa7c"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "498652c2d7f4c6a868c52fb33d5d8485"
server_key_iv: "713523c911e10f00608cfcad66d62580"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.931699 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:25.932346 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.932507 5328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:25.932588 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "509d7ce982d945c6b5311a3188e1fa7c"
format_stamp: "Formatted at 2026-09-27 21:07:25 on dist-test-slave-8bgs"
server_key: "498652c2d7f4c6a868c52fb33d5d8485"
server_key_iv: "713523c911e10f00608cfcad66d62580"
server_key_version: "encryptionkey@0"
I20260927 21:07:25.932641 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:25.935743 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:25.935935 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:25.936021 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:25.936172 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:25.936386 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:25.936414 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.936432 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:25.936445 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:25.942003 5328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41577
I20260927 21:07:25.942054 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41577 every 8 connection(s)
I20260927 21:07:25.942353 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:25.943192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5328
I20260927 21:07:25.943301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:25.943531 392 external_mini_cluster.cc:1428] Setting key 63ac78e8fddeec8242ef05991777aeaf
I20260927 21:07:25.945516 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.943143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60125 (local address 127.0.98.62:36335)
0927 21:07:25.943214 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.943216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.943445 (+ 229us) server_negotiation.cc:407] Connection header received
0927 21:07:25.944109 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.944110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.944116 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.944168 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.945350 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.945351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.945352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.945355 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.945366 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.945370 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.945414 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.945457 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.945468 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:07:25.945744 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.943037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36335 (local address 127.0.98.3:60125)
0927 21:07:25.943294 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.943337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.944002 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.944251 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.944255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.944265 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.945224 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:25.945225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.945251 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:25.945277 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.945412 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.945414 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.945416 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.945464 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.945507 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":154,"threads_started":1}
I20260927 21:07:25.946156 5456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36335
I20260927 21:07:25.946240 5456 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:25.946400 5456 heartbeater.cc:511] Master 127.0.98.62:36335 requested a full tablet report, sending...
I20260927 21:07:25.946717 5006 ts_manager.cc:194] Registered new tserver with Master: 509d7ce982d945c6b5311a3188e1fa7c (127.0.98.3:41577)
I20260927 21:07:25.955063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:07:25.957369 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.956312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.0.98.62:36335)
0927 21:07:25.956393 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.956395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.956466 (+ 71us) server_negotiation.cc:407] Connection header received
0927 21:07:25.956502 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.956503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.956506 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.956553 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.957142 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.957142 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.957143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.957145 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.957155 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.957158 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.957195 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.957239 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.957250 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:07:25.958899 5006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:25.960506 5006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:25.964141 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.962552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40509 (local address 127.0.0.1:44430)
0927 21:07:25.962720 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.962757 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.962853 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.963168 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.963174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.963180 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.963708 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:25.963709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.963724 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:25.963748 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.963975 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.963981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.963984 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.964057 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.964071 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260927 21:07:25.964366 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.962557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41577 (local address 127.0.0.1:59268)
0927 21:07:25.962821 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.962833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.962909 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.963705 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.963706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.963708 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.964016 (+ 308us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:25.964016 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.964023 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:25.964035 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.964253 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.964254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.964254 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.964300 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.964312 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260927 21:07:25.964457 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.962788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.0.98.2:40509)
0927 21:07:25.962979 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.962984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.962996 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:07:25.963023 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.963025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.963035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.963165 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.963836 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.963845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.963846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.963850 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.963884 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.963911 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.963981 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.964348 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.964364 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260927 21:07:25.964553 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.962583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38065 (local address 127.0.0.1:45254)
0927 21:07:25.962983 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:25.962991 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:25.963040 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:25.963229 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:25.963230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:25.963233 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:25.964039 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:25.964040 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:25.964046 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:25.964062 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:25.964404 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:25.964404 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:25.964405 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:25.964452 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:07:25.964467 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":66,"threads_started":1}
I20260927 21:07:25.964560 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.962629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.0.98.3:41577)
0927 21:07:25.963570 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.963574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.963583 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:25.963605 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.963607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.963615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.963688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.964110 (+ 422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.964116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.964116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.964120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.964151 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.964172 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.964227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.964452 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.964469 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":72,"threads_started":1}
I20260927 21:07:25.964804 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:25.962827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.0.98.1:38065)
0927 21:07:25.963001 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:25.963005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:25.963038 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:07:25.963119 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:25.963122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:25.963130 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:25.963225 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:25.964286 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:25.964291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:25.964291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:25.964296 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:25.964323 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:25.964339 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:25.964393 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:25.964709 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 21:07:25.964725 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":114,"threads_started":1}
I20260927 21:07:25.965204 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 87b233cee1344fb6b97f933497dc1b9a (DEFAULT_TABLE table=test-table [id=41dfd43521cd42c8a03ac6f598498227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:25.965569 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b233cee1344fb6b97f933497dc1b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.965526 5128 tablet_service.cc:1467] Processing CreateTablet for tablet 87b233cee1344fb6b97f933497dc1b9a (DEFAULT_TABLE table=test-table [id=41dfd43521cd42c8a03ac6f598498227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:25.965828 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b233cee1344fb6b97f933497dc1b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.967422 5390 tablet_service.cc:1467] Processing CreateTablet for tablet 87b233cee1344fb6b97f933497dc1b9a (DEFAULT_TABLE table=test-table [id=41dfd43521cd42c8a03ac6f598498227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:25.967757 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b233cee1344fb6b97f933497dc1b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:25.968870 5471 tablet_bootstrap.cc:492] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Bootstrap starting.
I20260927 21:07:25.969517 5471 tablet_bootstrap.cc:654] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.969823 5471 log.cc:824] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.970132 5470 tablet_bootstrap.cc:492] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: Bootstrap starting.
I20260927 21:07:25.970566 5471 tablet_bootstrap.cc:492] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: No bootstrap required, opened a new log
I20260927 21:07:25.970626 5471 ts_tablet_manager.cc:1397] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:25.970814 5470 tablet_bootstrap.cc:654] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.971107 5473 tablet_bootstrap.cc:492] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Bootstrap starting.
I20260927 21:07:25.971153 5470 log.cc:824] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.971742 5473 tablet_bootstrap.cc:654] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:25.971916 5470 tablet_bootstrap.cc:492] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: No bootstrap required, opened a new log
I20260927 21:07:25.971982 5470 ts_tablet_manager.cc:1397] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:07:25.972033 5473 log.cc:824] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:25.972774 5473 tablet_bootstrap.cc:492] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: No bootstrap required, opened a new log
I20260927 21:07:25.972870 5473 ts_tablet_manager.cc:1397] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:25.973438 5471 raft_consensus.cc:348] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:25.973619 5471 raft_consensus.cc:374] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.973675 5471 raft_consensus.cc:729] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1be1aabcfd46409259955bb773544f, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.973816 5471 consensus_queue.cc:260] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:25.974500 5470 raft_consensus.cc:348] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:25.974691 5470 raft_consensus.cc:374] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.974762 5470 raft_consensus.cc:729] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50d2ae251874f7c91b52b58fc831300, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.974908 5470 consensus_queue.cc:260] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:25.975212 5470 ts_tablet_manager.cc:1428] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:07:25.975328 5194 heartbeater.cc:503] Master 127.0.98.62:36335 was elected leader, sending a full tablet report...
I20260927 21:07:25.975489 5473 raft_consensus.cc:348] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:25.975637 5473 raft_consensus.cc:374] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:25.975675 5473 raft_consensus.cc:729] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509d7ce982d945c6b5311a3188e1fa7c, State: Initialized, Role: FOLLOWER
I20260927 21:07:25.975796 5473 consensus_queue.cc:260] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:25.976032 5473 ts_tablet_manager.cc:1428] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 21:07:25.976069 5456 heartbeater.cc:503] Master 127.0.98.62:36335 was elected leader, sending a full tablet report...
I20260927 21:07:25.976100 5471 ts_tablet_manager.cc:1428] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 21:07:25.976217 5325 heartbeater.cc:503] Master 127.0.98.62:36335 was elected leader, sending a full tablet report...
W20260927 21:07:26.078361 5326 tablet.cc:2369] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:07:26.192874 5457 tablet.cc:2369] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:26.200702 5476 raft_consensus.cc:482] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:26.200801 5476 raft_consensus.cc:504] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:26.201102 5476 leader_election.cc:288] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 509d7ce982d945c6b5311a3188e1fa7c (127.0.98.3:41577), a50d2ae251874f7c91b52b58fc831300 (127.0.98.1:38065)
I20260927 21:07:26.207674 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.201371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38319 (local address 127.0.98.1:38065)
0927 21:07:26.201420 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.201423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.201436 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:07:26.201468 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.201471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.201481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.201545 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:26.207397 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:26.207399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:26.207399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:26.207405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:26.207426 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:26.207431 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:26.207498 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:26.207573 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:07:26.207589 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:07:26.207697 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.201309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38065 (local address 127.0.98.2:38319)
0927 21:07:26.201364 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:26.201377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:26.201417 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:26.201570 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:26.201571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:26.201573 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:26.207270 (+ 5697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:26.207271 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:26.207281 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:26.207296 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:26.207505 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:26.207506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:26.207507 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:26.207560 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:07:26.207581 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:07:26.207720 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.201384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54721 (local address 127.0.98.3:41577)
0927 21:07:26.201423 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.201426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.201585 (+ 159us) server_negotiation.cc:407] Connection header received
0927 21:07:26.201615 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.201616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.201622 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.201675 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:26.207397 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:26.207399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:26.207399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:26.207404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:26.207441 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:26.207446 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:26.207499 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:26.207637 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:07:26.207654 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:26.207728 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.201309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41577 (local address 127.0.98.2:54721)
0927 21:07:26.201497 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:26.201504 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:26.201564 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:26.201697 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:26.201698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:26.201700 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:26.207270 (+ 5570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:26.207271 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:26.207281 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:26.207296 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:26.207554 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:26.207555 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:26.207556 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:26.207611 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:07:26.207624 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":12,"thread_start_us":92,"threads_started":1}
I20260927 21:07:26.208036 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b233cee1344fb6b97f933497dc1b9a" candidate_uuid: "2d1be1aabcfd46409259955bb773544f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509d7ce982d945c6b5311a3188e1fa7c" is_pre_election: true
I20260927 21:07:26.208073 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b233cee1344fb6b97f933497dc1b9a" candidate_uuid: "2d1be1aabcfd46409259955bb773544f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50d2ae251874f7c91b52b58fc831300" is_pre_election: true
I20260927 21:07:26.208199 5410 raft_consensus.cc:2454] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d1be1aabcfd46409259955bb773544f in term 0.
I20260927 21:07:26.208230 5148 raft_consensus.cc:2454] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d1be1aabcfd46409259955bb773544f in term 0.
I20260927 21:07:26.208462 5212 leader_election.cc:302] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d1be1aabcfd46409259955bb773544f, 509d7ce982d945c6b5311a3188e1fa7c; no voters:
I20260927 21:07:26.208585 5476 raft_consensus.cc:2781] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:26.208643 5476 raft_consensus.cc:482] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:26.208678 5476 raft_consensus.cc:3037] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:26.209358 5476 raft_consensus.cc:504] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:26.209499 5476 leader_election.cc:288] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [CANDIDATE]: Term 1 election: Requested vote from peers 509d7ce982d945c6b5311a3188e1fa7c (127.0.98.3:41577), a50d2ae251874f7c91b52b58fc831300 (127.0.98.1:38065)
I20260927 21:07:26.209651 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b233cee1344fb6b97f933497dc1b9a" candidate_uuid: "2d1be1aabcfd46409259955bb773544f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "509d7ce982d945c6b5311a3188e1fa7c"
I20260927 21:07:26.209677 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b233cee1344fb6b97f933497dc1b9a" candidate_uuid: "2d1be1aabcfd46409259955bb773544f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50d2ae251874f7c91b52b58fc831300"
I20260927 21:07:26.209730 5410 raft_consensus.cc:3037] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:26.209750 5148 raft_consensus.cc:3037] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:26.210407 5148 raft_consensus.cc:2454] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d1be1aabcfd46409259955bb773544f in term 1.
I20260927 21:07:26.210426 5410 raft_consensus.cc:2454] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d1be1aabcfd46409259955bb773544f in term 1.
I20260927 21:07:26.210608 5212 leader_election.cc:302] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d1be1aabcfd46409259955bb773544f, 509d7ce982d945c6b5311a3188e1fa7c; no voters:
I20260927 21:07:26.210742 5476 raft_consensus.cc:2781] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:26.210925 5476 raft_consensus.cc:686] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 1 LEADER]: Becoming Leader. State: Replica: 2d1be1aabcfd46409259955bb773544f, State: Running, Role: LEADER
I20260927 21:07:26.211022 5476 consensus_queue.cc:237] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } }
I20260927 21:07:26.211902 5005 catalog_manager.cc:5495] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d1be1aabcfd46409259955bb773544f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2d1be1aabcfd46409259955bb773544f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d1be1aabcfd46409259955bb773544f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 } health_report { overall_health: UNKNOWN } } }
W20260927 21:07:26.214349 5195 tablet.cc:2369] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:26.246946 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.245879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.0.98.2:40509)
0927 21:07:26.245950 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.245952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.245961 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:26.246024 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.246025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.246045 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.246112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:26.246733 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:26.246734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:26.246735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:26.246740 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:26.246756 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:26.246761 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:26.246813 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:26.246859 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 21:07:26.246873 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:07:26.248745 5410 raft_consensus.cc:1261] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 1 FOLLOWER]: Refusing update from remote peer 2d1be1aabcfd46409259955bb773544f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:26.248746 5148 raft_consensus.cc:1261] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300 [term 1 FOLLOWER]: Refusing update from remote peer 2d1be1aabcfd46409259955bb773544f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:26.249101 5480 consensus_queue.cc:1035] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a50d2ae251874f7c91b52b58fc831300" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:26.249231 5476 consensus_queue.cc:1035] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [LEADER]: Connected to new peer: Peer: permanent_uuid: "509d7ce982d945c6b5311a3188e1fa7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:26.253840 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:26.253942 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:26.254653 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17210538af314cb3bef013431b69c142: Sending DeleteTablet for 3 replicas of tablet 87b233cee1344fb6b97f933497dc1b9a
I20260927 21:07:26.254956 5259 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b233cee1344fb6b97f933497dc1b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:26 UTC) from {username='slave'} at 127.0.0.1:44430
I20260927 21:07:26.255023 5390 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b233cee1344fb6b97f933497dc1b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:26 UTC) from {username='slave'} at 127.0.0.1:59268
I20260927 21:07:26.255185 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b233cee1344fb6b97f933497dc1b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:26 UTC) from {username='slave'} at 127.0.0.1:45254
I20260927 21:07:26.255239 5494 tablet_replica.cc:330] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: stopping tablet replica
I20260927 21:07:26.255329 5494 raft_consensus.cc:2229] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:26.255410 5495 tablet_replica.cc:330] T 87b233cee1344fb6b97f933497dc1b9a P a50d2ae251874f7c91b52b58fc831300: stopping tablet replica
I20260927 21:07:26.255430 5494 raft_consensus.cc:2258] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:26.255611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5066
I20260927 21:07:26.255643 5493 tablet_replica.cc:330] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: stopping tablet replica
I20260927 21:07:26.255730 5493 raft_consensus.cc:2229] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:26.255837 5493 raft_consensus.cc:2258] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:26.256875 5493 ts_tablet_manager.cc:1905] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:26.258352 5493 ts_tablet_manager.cc:1918] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:26.258442 5493 log.cc:1196] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/87b233cee1344fb6b97f933497dc1b9a
I20260927 21:07:26.258693 5493 ts_tablet_manager.cc:1939] T 87b233cee1344fb6b97f933497dc1b9a P 2d1be1aabcfd46409259955bb773544f: Deleting consensus metadata
I20260927 21:07:26.259055 4993 catalog_manager.cc:4841] TS 2d1be1aabcfd46409259955bb773544f (127.0.98.2:40509): tablet 87b233cee1344fb6b97f933497dc1b9a (table test-table [id=41dfd43521cd42c8a03ac6f598498227]) successfully deleted
I20260927 21:07:26.260174 5494 ts_tablet_manager.cc:1905] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:26.261757 5494 ts_tablet_manager.cc:1918] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:26.261840 5494 log.cc:1196] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/87b233cee1344fb6b97f933497dc1b9a
I20260927 21:07:26.262107 5494 ts_tablet_manager.cc:1939] T 87b233cee1344fb6b97f933497dc1b9a P 509d7ce982d945c6b5311a3188e1fa7c: Deleting consensus metadata
W20260927 21:07:26.262511 4991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38065 (error 108)
I20260927 21:07:26.262526 4990 catalog_manager.cc:4841] TS 509d7ce982d945c6b5311a3188e1fa7c (127.0.98.3:41577): tablet 87b233cee1344fb6b97f933497dc1b9a (table test-table [id=41dfd43521cd42c8a03ac6f598498227]) successfully deleted
I20260927 21:07:26.262769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5197
I20260927 21:07:26.263439 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.262890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38065 (local address 127.0.0.1:45262)
0927 21:07:26.263347 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:26.263388 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
W20260927 21:07:26.263578 4991 catalog_manager.cc:4584] TS a50d2ae251874f7c91b52b58fc831300 (127.0.98.1:38065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87b233cee1344fb6b97f933497dc1b9a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38065: connect: Connection refused (error 111)
I20260927 21:07:26.268323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5328
I20260927 21:07:26.272951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 4975
2026-09-27T21:07:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (848 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 21:07:26.291855 392 mini_oidc.cc:152] Starting JWKS server
I20260927 21:07:26.292837 392 webserver.cc:460] Webserver started at https://127.0.0.1:46125/ using document root <none> and password file <none>
I20260927 21:07:26.292912 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:07:26.293042 392 webserver.cc:460] Webserver started at http://127.0.0.1:33169/ using document root <none> and password file <none>
2026-09-27T21:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:26Z Disabled control of system clock
I20260927 21:07:26.300510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39739
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39739
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/testchainca.cert with env {}
W20260927 21:07:26.380162 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:26.380364 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:26.380386 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:26.381500 5502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:26.381536 5502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:26.381548 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:26.381560 5502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:26.381572 5502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:07:26.381584 5502 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:07:26.381603 5502 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46125/jwks.json
I20260927 21:07:26.383061 5502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:26.383308 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:26.383488 5502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:26.388201 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:26.388306 5502 server_base.cc:1034] running on GCE node
W20260927 21:07:26.388207 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:26.388180 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:26.388684 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:26.389000 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:26.390168 5502 hybrid_clock.cc:648] HybridClock initialized: now 1790543246390163 us; error 36 us; skew 500 ppm
I20260927 21:07:26.391374 5502 webserver.cc:460] Webserver started at http://127.0.98.62:32785/ using document root <none> and password file <none>
I20260927 21:07:26.391592 5502 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:26.391660 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:26.391803 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:26.392881 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "0a9e50f9a48446b08a9eed490c4c7e88"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "4cd7e90d896b4f055aff65c38294a678"
server_key_iv: "63a06297fec60029ab0edccd51856ef2"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.393244 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a9e50f9a48446b08a9eed490c4c7e88"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "4cd7e90d896b4f055aff65c38294a678"
server_key_iv: "63a06297fec60029ab0edccd51856ef2"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.394542 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:26.395387 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:26.395543 5502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:26.395644 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "0a9e50f9a48446b08a9eed490c4c7e88"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "4cd7e90d896b4f055aff65c38294a678"
server_key_iv: "63a06297fec60029ab0edccd51856ef2"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.395715 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:26.403759 5502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:26.404021 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:26.404168 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:26.407720 5502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39739
I20260927 21:07:26.407753 5568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39739 every 8 connection(s)
I20260927 21:07:26.408056 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:26.408981 5569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:26.411185 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88: Bootstrap starting.
I20260927 21:07:26.411769 5569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:26.412063 5569 log.cc:824] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:26.412693 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88: No bootstrap required, opened a new log
I20260927 21:07:26.414254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5502
I20260927 21:07:26.414338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:26.414265 5569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } }
I20260927 21:07:26.414464 5569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:26.414541 392 external_mini_cluster.cc:1428] Setting key 66fdc327a341652f70d54fe9a8be8c52
I20260927 21:07:26.414535 5569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9e50f9a48446b08a9eed490c4c7e88, State: Initialized, Role: FOLLOWER
I20260927 21:07:26.414671 5569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } }
I20260927 21:07:26.414783 5569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:26.414819 5569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:26.414873 5569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:26.415609 5569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } }
I20260927 21:07:26.415714 5569 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a9e50f9a48446b08a9eed490c4c7e88; no voters:
I20260927 21:07:26.415881 5569 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:26.415946 5574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:26.416086 5574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [term 1 LEADER]: Becoming Leader. State: Replica: 0a9e50f9a48446b08a9eed490c4c7e88, State: Running, Role: LEADER
I20260927 21:07:26.416169 5574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } }
I20260927 21:07:26.416196 5569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:26.416476 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a9e50f9a48446b08a9eed490c4c7e88. Latest consensus state: current_term: 1 leader_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } }
I20260927 21:07:26.416564 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:26.416486 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9e50f9a48446b08a9eed490c4c7e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } }
I20260927 21:07:26.416713 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:26.416831 5579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:26.417374 5579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:26.419601 5579 catalog_manager.cc:1347] Generated new cluster ID: 36086836c6bb4f1e888286e39dc52754
I20260927 21:07:26.419657 5579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:26.420711 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.414968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.0.98.62:39739)
0927 21:07:26.415208 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.415217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.415253 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:07:26.415392 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.415399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.415483 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.415639 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:26.415873 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.417684 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.418527 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.419031 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.419211 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.419937 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:26.419940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:26.419941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:26.420134 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:26.420175 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:26.420187 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:26.420251 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:26.420353 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:07:26.420509 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":108,"threads_started":1}
I20260927 21:07:26.421563 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:07:26.427250 5579 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:26.427745 5579 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:26.435459 5579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a9e50f9a48446b08a9eed490c4c7e88: Generated new TSK 0
I20260927 21:07:26.435631 5579 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:07:26.456462 5599 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 21:07:26.456578 5599 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 21:07:26.457079 5598 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:26.440957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39739 (local address 127.0.0.1:51278)
0927 21:07:26.441160 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:26.441171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:26.441224 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:26.441385 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:26.441385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:26.441389 (+ 4us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:07:26.441446 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:26.441447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.442350 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.442351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:26.442925 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:26.442927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.443388 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.443391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:26.443452 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.443469 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:07:26.456694 (+ 13225us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:07:26.456783 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39739: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:39739: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 21:07:26.457446 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.441045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.0.98.62:39739)
0927 21:07:26.441099 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.441101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.441226 (+ 125us) server_negotiation.cc:407] Connection header received
0927 21:07:26.441293 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.441293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.441305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.441383 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:07:26.441503 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.442275 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.443001 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.443321 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.443389 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.443570 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:07:26.456567 (+ 12997us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:07:26.457287 (+ 720us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 21:07:26.457489 5573 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260927 21:07:26.457475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5502
2026-09-27T21:07:26Z chronyd exiting
I20260927 21:07:26.493364 392 test_util.cc:175] -----------------------------------------------
I20260927 21:07:26.493458 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790543220945705-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (202 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 21:07:26.494190 392 mini_oidc.cc:152] Starting JWKS server
I20260927 21:07:26.494894 392 webserver.cc:460] Webserver started at https://127.0.0.1:46787/ using document root <none> and password file <none>
I20260927 21:07:26.494952 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:07:26.495127 392 webserver.cc:460] Webserver started at http://127.0.0.1:39259/ using document root <none> and password file <none>
2026-09-27T21:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:26Z Disabled control of system clock
I20260927 21:07:26.502286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43501
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46787/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/testchainca.cert with env {}
W20260927 21:07:26.579425 5605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:26.579615 5605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:26.579636 5605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:26.580726 5605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:26.580760 5605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:26.580773 5605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:26.580786 5605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:26.580797 5605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:07:26.580811 5605 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:07:26.580826 5605 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46787/jwks.json
I20260927 21:07:26.582227 5605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46787/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:26.582404 5605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:26.582553 5605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:26.586975 5613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:26.587136 5605 server_base.cc:1034] running on GCE node
W20260927 21:07:26.586970 5610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:26.587056 5611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:26.587488 5605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:26.587781 5605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:26.588941 5605 hybrid_clock.cc:648] HybridClock initialized: now 1790543246588930 us; error 41 us; skew 500 ppm
I20260927 21:07:26.590080 5605 webserver.cc:460] Webserver started at http://127.0.98.62:42137/ using document root <none> and password file <none>
I20260927 21:07:26.590314 5605 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:26.590381 5605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:26.590526 5605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:26.591564 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "af549d1f816746fe884e9f1879fff50d"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "f9822aeeb54f6cafbb5e6fe033f4702e"
server_key_iv: "48584e1be7018fd8f1f744c69df2fd5c"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.591912 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "af549d1f816746fe884e9f1879fff50d"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "f9822aeeb54f6cafbb5e6fe033f4702e"
server_key_iv: "48584e1be7018fd8f1f744c69df2fd5c"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.593128 5605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:26.593763 5619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:26.593930 5605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:26.594025 5605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "af549d1f816746fe884e9f1879fff50d"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "f9822aeeb54f6cafbb5e6fe033f4702e"
server_key_iv: "48584e1be7018fd8f1f744c69df2fd5c"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.594099 5605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:26.604398 5605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:26.604648 5605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:26.604789 5605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:26.608430 5605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43501
I20260927 21:07:26.608474 5671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43501 every 8 connection(s)
I20260927 21:07:26.608800 5605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:26.609706 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:26.611917 5672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d: Bootstrap starting.
I20260927 21:07:26.612471 5672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:26.612756 5672 log.cc:824] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:26.613386 5672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d: No bootstrap required, opened a new log
I20260927 21:07:26.614904 5672 raft_consensus.cc:348] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af549d1f816746fe884e9f1879fff50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260927 21:07:26.615033 5672 raft_consensus.cc:374] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:26.615082 5672 raft_consensus.cc:729] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af549d1f816746fe884e9f1879fff50d, State: Initialized, Role: FOLLOWER
I20260927 21:07:26.615279 5672 consensus_queue.cc:260] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af549d1f816746fe884e9f1879fff50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260927 21:07:26.615363 5672 raft_consensus.cc:388] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:26.615407 5672 raft_consensus.cc:482] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:26.615454 5672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:26.615950 5672 raft_consensus.cc:504] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af549d1f816746fe884e9f1879fff50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260927 21:07:26.616065 5672 leader_election.cc:302] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af549d1f816746fe884e9f1879fff50d; no voters:
I20260927 21:07:26.616214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5605
I20260927 21:07:26.616246 5672 leader_election.cc:288] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:26.616303 5675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:26.616323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:26.616521 392 external_mini_cluster.cc:1428] Setting key d3a800c49f654685917445ca19de5a04
I20260927 21:07:26.616487 5675 raft_consensus.cc:686] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [term 1 LEADER]: Becoming Leader. State: Replica: af549d1f816746fe884e9f1879fff50d, State: Running, Role: LEADER
I20260927 21:07:26.616487 5672 sys_catalog.cc:564] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:26.616621 5675 consensus_queue.cc:237] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af549d1f816746fe884e9f1879fff50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260927 21:07:26.616923 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af549d1f816746fe884e9f1879fff50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af549d1f816746fe884e9f1879fff50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } } }
I20260927 21:07:26.616992 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:26.617274 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader af549d1f816746fe884e9f1879fff50d. Latest consensus state: current_term: 1 leader_uuid: "af549d1f816746fe884e9f1879fff50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af549d1f816746fe884e9f1879fff50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } } }
I20260927 21:07:26.617390 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:26.617543 5681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:26.618052 5681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:26.621300 5681 catalog_manager.cc:1347] Generated new cluster ID: 825fe5f72a2944ed9479228b039436f6
I20260927 21:07:26.621351 5681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:26.623974 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.616967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.0.98.62:43501)
0927 21:07:26.617562 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.617570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.617603 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:07:26.617727 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.617732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.617780 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.617937 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:26.618143 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.619297 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.620381 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.620856 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.620993 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.623326 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:26.623330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:26.623331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:26.623510 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:26.623540 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:26.623550 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:26.623628 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:26.623713 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:07:26.623790 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":75,"threads_started":1}
I20260927 21:07:26.624886 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:07:26.640695 5681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:26.641352 5681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:26.643968 5681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af549d1f816746fe884e9f1879fff50d: Generated new TSK 0
I20260927 21:07:26.644116 5681 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:07:26.665190 5702 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 21:07:26.665273 5702 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:07:26.665582 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.651487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.98.62:43501)
0927 21:07:26.651543 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.651545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.651675 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:07:26.651723 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.651724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.651733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.651788 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:07:26.651920 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.652391 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.653477 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.653663 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.653715 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.653898 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:07:26.665414 (+ 11516us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:07:26.665535 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 21:07:26.665647 5680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 21:07:26.665743 5701 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:26.651439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43501 (local address 127.0.0.1:36428)
0927 21:07:26.651590 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:26.651602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:26.651666 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:26.651809 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:26.651810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:26.651812 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:07:26.651861 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:26.651863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.652477 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.652478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:26.653400 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:26.653402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.653743 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.653746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:26.653807 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.653813 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:07:26.665535 (+ 11722us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:07:26.665643 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46787/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260927 21:07:26.666199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5605
2026-09-27T21:07:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (208 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 21:07:26.703157 392 mini_oidc.cc:152] Starting JWKS server
I20260927 21:07:26.703912 392 webserver.cc:460] Webserver started at https://127.0.0.1:37471/ using document root <none> and password file <none>
I20260927 21:07:26.703982 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:07:26.704160 392 webserver.cc:460] Webserver started at http://127.0.0.1:46639/ using document root <none> and password file <none>
2026-09-27T21:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:26Z Disabled control of system clock
I20260927 21:07:26.711620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36977
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:07:26.790112 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:26.790315 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:26.790336 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:26.791436 5708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:26.791471 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:26.791483 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:26.791496 5708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:26.791508 5708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:07:26.791522 5708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:07:26.791538 5708 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37471/jwks.json
I20260927 21:07:26.792811 5708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:26.792995 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:26.793222 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:26.797619 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:26.797662 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:26.797638 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:26.797917 5708 server_base.cc:1034] running on GCE node
I20260927 21:07:26.798173 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:26.798453 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:26.799605 5708 hybrid_clock.cc:648] HybridClock initialized: now 1790543246799590 us; error 27 us; skew 500 ppm
I20260927 21:07:26.800782 5708 webserver.cc:460] Webserver started at http://127.0.98.62:45601/ using document root <none> and password file <none>
I20260927 21:07:26.801012 5708 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:26.801057 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:26.801229 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:26.802237 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "324634b4f4fb4e17b3f54d2bfd0b0706"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "5e818dc646bb1e3a71c15cbdf2de7080"
server_key_iv: "2da4a65ecdf5bb9fe2818d3128f565a6"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.802590 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "324634b4f4fb4e17b3f54d2bfd0b0706"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "5e818dc646bb1e3a71c15cbdf2de7080"
server_key_iv: "2da4a65ecdf5bb9fe2818d3128f565a6"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.803938 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:26.804607 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:26.804816 5708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:26.804926 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "324634b4f4fb4e17b3f54d2bfd0b0706"
format_stamp: "Formatted at 2026-09-27 21:07:26 on dist-test-slave-8bgs"
server_key: "5e818dc646bb1e3a71c15cbdf2de7080"
server_key_iv: "2da4a65ecdf5bb9fe2818d3128f565a6"
server_key_version: "encryptionkey@0"
I20260927 21:07:26.805001 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:26.825555 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:26.825783 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:26.825878 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:26.829578 5708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36977
I20260927 21:07:26.829607 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36977 every 8 connection(s)
I20260927 21:07:26.829944 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:26.830523 5775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:26.832882 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706: Bootstrap starting.
I20260927 21:07:26.833426 5775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:26.833707 5775 log.cc:824] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:26.834336 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706: No bootstrap required, opened a new log
I20260927 21:07:26.835479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5708
I20260927 21:07:26.835574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:26.835777 392 external_mini_cluster.cc:1428] Setting key 74aba7ec6c9134105beb7697d8f45aaa
I20260927 21:07:26.836302 5775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260927 21:07:26.836486 5775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:26.836535 5775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324634b4f4fb4e17b3f54d2bfd0b0706, State: Initialized, Role: FOLLOWER
I20260927 21:07:26.836658 5775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260927 21:07:26.836751 5775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:26.836789 5775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:26.836833 5775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:26.837333 5775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260927 21:07:26.837445 5775 leader_election.cc:302] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324634b4f4fb4e17b3f54d2bfd0b0706; no voters:
I20260927 21:07:26.837610 5775 leader_election.cc:288] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:26.837641 5780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:26.837796 5780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [term 1 LEADER]: Becoming Leader. State: Replica: 324634b4f4fb4e17b3f54d2bfd0b0706, State: Running, Role: LEADER
I20260927 21:07:26.837852 5775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:26.837885 5780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } }
I20260927 21:07:26.838194 5780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324634b4f4fb4e17b3f54d2bfd0b0706. Latest consensus state: current_term: 1 leader_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } } }
I20260927 21:07:26.838266 5780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:26.838207 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324634b4f4fb4e17b3f54d2bfd0b0706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36977 } } }
I20260927 21:07:26.838306 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:26.839066 5786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:26.839608 5786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:26.841243 5786 catalog_manager.cc:1347] Generated new cluster ID: 0cfaaaaeba524d8da79b4d515c31df3a
I20260927 21:07:26.841296 5786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:26.841817 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.836230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.0.98.62:36977)
0927 21:07:26.836454 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.836461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.836483 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:07:26.836606 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.836613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.836681 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.836815 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:26.837039 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.838652 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.839778 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.840334 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.840483 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.841126 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:26.841130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:26.841130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:26.841322 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:26.841362 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:26.841376 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:26.841449 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:26.841541 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:07:26.841616 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260927 21:07:26.842727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:07:26.859444 5786 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:26.860018 5786 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:26.876104 5786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 324634b4f4fb4e17b3f54d2bfd0b0706: Generated new TSK 0
I20260927 21:07:26.876274 5786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:26.893532 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:26.890083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.62:36977)
0927 21:07:26.890142 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:26.890144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:26.890239 (+ 95us) server_negotiation.cc:407] Connection header received
0927 21:07:26.890307 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:26.890308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:26.890318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:26.890376 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:07:26.890511 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.890970 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.892121 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.892299 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.892346 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.892547 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:07:26.893315 (+ 768us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:07:26.893461 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 21:07:26.893596 5779 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 21:07:26.893607 5804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:26.890006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36977 (local address 127.0.0.1:54432)
0927 21:07:26.890166 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:26.890177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:26.890242 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:26.890393 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:26.890394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:26.890396 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:07:26.890453 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:26.890455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.891059 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.891061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:26.892030 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:26.892034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:26.892370 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:26.892374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:26.892444 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:26.892450 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:07:26.893414 (+ 964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:07:26.893502 (+ 88us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260927 21:07:26.893981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5708
E20260927 21:07:26.904685 5805 webserver.cc:533] Webserver: sslize failed:
E20260927 21:07:26.904738 5805 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T21:07:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T21:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:26Z Disabled control of system clock
I20260927 21:07:26.938046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40419
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40419
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/testchainca.cert with env {}
W20260927 21:07:27.015729 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:27.015947 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:27.015985 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:27.017119 5809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:27.017182 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:27.017203 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:27.017225 5809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:27.017267 5809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:07:27.017309 5809 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:07:27.017335 5809 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 21:07:27.018738 5809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:27.018990 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:27.019250 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:27.023731 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:27.023725 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:27.023742 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:27.024091 5809 server_base.cc:1034] running on GCE node
I20260927 21:07:27.024340 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:27.024603 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:27.025763 5809 hybrid_clock.cc:648] HybridClock initialized: now 1790543247025757 us; error 34 us; skew 500 ppm
I20260927 21:07:27.026998 5809 webserver.cc:460] Webserver started at http://127.0.98.62:42203/ using document root <none> and password file <none>
I20260927 21:07:27.027259 5809 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:27.027330 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:27.027456 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:27.028468 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a"
format_stamp: "Formatted at 2026-09-27 21:07:27 on dist-test-slave-8bgs"
server_key: "f9912065305420d83d3d53e9c39eecd5"
server_key_iv: "30f4a9523dc61d05967fd03b94b6fcb1"
server_key_version: "encryptionkey@0"
I20260927 21:07:27.028817 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a"
format_stamp: "Formatted at 2026-09-27 21:07:27 on dist-test-slave-8bgs"
server_key: "f9912065305420d83d3d53e9c39eecd5"
server_key_iv: "30f4a9523dc61d05967fd03b94b6fcb1"
server_key_version: "encryptionkey@0"
I20260927 21:07:27.030021 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:27.030725 5823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:27.030895 5809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:27.030988 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a"
format_stamp: "Formatted at 2026-09-27 21:07:27 on dist-test-slave-8bgs"
server_key: "f9912065305420d83d3d53e9c39eecd5"
server_key_iv: "30f4a9523dc61d05967fd03b94b6fcb1"
server_key_version: "encryptionkey@0"
I20260927 21:07:27.031070 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:27.042891 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:27.043247 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:27.043401 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:27.047214 5809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40419
I20260927 21:07:27.047247 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40419 every 8 connection(s)
I20260927 21:07:27.047575 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:27.048484 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:27.050629 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a: Bootstrap starting.
I20260927 21:07:27.051231 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:27.051482 5876 log.cc:824] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:27.051837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5809
I20260927 21:07:27.051942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:27.052161 392 external_mini_cluster.cc:1428] Setting key d3bb0a4f1a7e0af2171779c3e9b4c6ff
I20260927 21:07:27.052193 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a: No bootstrap required, opened a new log
I20260927 21:07:27.053969 5876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 21:07:27.054131 5876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:27.054178 5876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7f6c4d11c44d5facb5c7a95a31db9a, State: Initialized, Role: FOLLOWER
I20260927 21:07:27.054313 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 21:07:27.054426 5876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:27.054471 5876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:27.054522 5876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:27.055050 5876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 21:07:27.055212 5876 leader_election.cc:302] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7f6c4d11c44d5facb5c7a95a31db9a; no voters:
I20260927 21:07:27.055390 5876 leader_election.cc:288] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:27.055413 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:27.055519 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [term 1 LEADER]: Becoming Leader. State: Replica: 8b7f6c4d11c44d5facb5c7a95a31db9a, State: Running, Role: LEADER
I20260927 21:07:27.055598 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 21:07:27.055642 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:27.055898 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } } }
I20260927 21:07:27.056034 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7f6c4d11c44d5facb5c7a95a31db9a. Latest consensus state: current_term: 1 leader_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7f6c4d11c44d5facb5c7a95a31db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } } }
I20260927 21:07:27.056105 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:27.056118 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:27.056463 5890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:27.056946 5890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:27.058640 5890 catalog_manager.cc:1347] Generated new cluster ID: fa58c26c04ae48ea94731e64350c5fc1
I20260927 21:07:27.058712 5890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:27.059274 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:27.052636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.98.62:40419)
0927 21:07:27.052834 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:27.052841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:27.052863 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:07:27.053952 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:27.053958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:27.054011 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:27.054117 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:27.054356 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:27.055890 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:27.057065 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:27.057674 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:27.057822 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:27.058543 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:27.058547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:27.058547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:27.058761 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:27.058803 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:27.058816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:27.058882 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:27.058975 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 21:07:27.059029 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260927 21:07:27.060108 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:07:27.079546 5890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:27.080036 5890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:27.087445 5890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b7f6c4d11c44d5facb5c7a95a31db9a: Generated new TSK 0
I20260927 21:07:27.087610 5890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:27.214323 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:27.094136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.0.98.62:40419)
0927 21:07:27.094177 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:27.094180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:27.094302 (+ 122us) server_negotiation.cc:407] Connection header received
0927 21:07:27.094385 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:27.094386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:27.094400 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:27.094461 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:07:27.094583 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:27.095085 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:27.095912 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:27.096117 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:27.096190 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:27.096383 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:07:27.214105 (+117722us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:07:27.214265 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50288: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:07:27.214411 5880 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50288: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 21:07:27.214438 5905 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:27.094069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40419 (local address 127.0.0.1:50288)
0927 21:07:27.094214 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:27.094224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:27.094320 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:27.094458 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:27.094459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:27.094461 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:07:27.094521 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:27.094524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:27.095192 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:27.095193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:27.095813 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:27.095815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:27.096209 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:27.096213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:27.096268 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:27.096274 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:07:27.214238 (+117964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:07:27.214355 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260927 21:07:27.214890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5809
2026-09-27T21:07:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (301 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:27 dist-test-slave-8bgs krb5kdc[5917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:27 dist-test-slave-8bgs krb5kdc[5917](info): set up 2 sockets
Sep 27 21:07:27 dist-test-slave-8bgs krb5kdc[5917](info): commencing operation
krb5kdc: starting...
W20260927 21:07:28.411257 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:28 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:28.537467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:28.615377 5930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:28.615598 5930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:28.615633 5930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:28.616746 5930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:28.616819 5930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:28.616844 5930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:28.616874 5930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:28.616897 5930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:28.618330 5930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:28.618600 5930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:28.618805 5930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:28.623592 5930 server_base.cc:1034] running on GCE node
W20260927 21:07:28.623559 5938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:28.623559 5936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:28.623546 5935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:28.624030 5930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:28.624337 5930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:28.625509 5930 hybrid_clock.cc:648] HybridClock initialized: now 1790543248625495 us; error 29 us; skew 500 ppm
Sep 27 21:07:28 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:28.627872 5930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:28.628326 5930 webserver.cc:460] Webserver started at http://127.0.98.62:46005/ using document root <none> and password file <none>
I20260927 21:07:28.628556 5930 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:28.628621 5930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:28.628765 5930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:28.629756 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "1a1cdf3c7358aba74a53d4d214eb9283"
server_key_iv: "64a84bd5d614812153f73b677327a56b"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.630115 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "1a1cdf3c7358aba74a53d4d214eb9283"
server_key_iv: "64a84bd5d614812153f73b677327a56b"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.631423 5930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:28.632108 5945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:28.632266 5930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:28.632373 5930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "1a1cdf3c7358aba74a53d4d214eb9283"
server_key_iv: "64a84bd5d614812153f73b677327a56b"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.632447 5930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:28.645666 5930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:28.647387 5930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:28.647554 5930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:28.651217 5930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43295
I20260927 21:07:28.651258 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43295 every 8 connection(s)
I20260927 21:07:28.651733 5930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:28.652328 5998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:28.654493 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac: Bootstrap starting.
I20260927 21:07:28.655057 5998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:28.655390 5998 log.cc:824] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:28.656014 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac: No bootstrap required, opened a new log
I20260927 21:07:28.657548 5998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260927 21:07:28.657682 5998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:28.657706 5998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fbae58d8eb4ab6bdac1653cb4cc8ac, State: Initialized, Role: FOLLOWER
I20260927 21:07:28.657804 5998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260927 21:07:28.657907 5998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:28.657974 5998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:28.658028 5998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:28.658915 5998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260927 21:07:28.659029 5998 leader_election.cc:302] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06fbae58d8eb4ab6bdac1653cb4cc8ac; no voters:
I20260927 21:07:28.659233 5998 leader_election.cc:288] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:28.659281 6001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:28.659438 6001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [term 1 LEADER]: Becoming Leader. State: Replica: 06fbae58d8eb4ab6bdac1653cb4cc8ac, State: Running, Role: LEADER
I20260927 21:07:28.659492 5998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:28.659539 6001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } }
I20260927 21:07:28.659902 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06fbae58d8eb4ab6bdac1653cb4cc8ac. Latest consensus state: current_term: 1 leader_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } } }
I20260927 21:07:28.660000 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:28.659893 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fbae58d8eb4ab6bdac1653cb4cc8ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43295 } } }
I20260927 21:07:28.660140 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:28.660457 6009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:28.660959 6009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:28.661616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 5930
I20260927 21:07:28.661746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:28.661955 392 external_mini_cluster.cc:1428] Setting key 3036f5165972818d6079fef83ec1b8a9
I20260927 21:07:28.662544 6009 catalog_manager.cc:1347] Generated new cluster ID: 3d5f8538515f497e86ade61ba2624ca3
I20260927 21:07:28.662596 6009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:28 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:28.672004 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:28.662422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.0.98.62:43295)
0927 21:07:28.662554 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:28.662562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:28.663369 (+ 807us) server_negotiation.cc:407] Connection header received
0927 21:07:28.663535 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:28.663540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:28.663590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:28.663723 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:28.664334 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:28.665229 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:28.665971 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:28.666392 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:28.666514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:28.668846 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:28.668849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:28.668850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:28.668861 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:28.670500 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:28.670880 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:28.670883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:28.670883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:28.670970 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:28.671188 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:28.671189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:28.671189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:28.671413 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:28.671586 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:28.671662 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:07:28.671799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260927 21:07:28.674543 6009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:28.675230 6009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:28.689757 6009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06fbae58d8eb4ab6bdac1653cb4cc8ac: Generated new TSK 0
I20260927 21:07:28.689930 6009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:28.715590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:28.827603 6024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:28.827798 6024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:28.827822 6024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:28.829030 6024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:28.829070 6024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:28.829097 6024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:28.830612 6024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:28.830857 6024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:28.831071 6024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:28.836604 6029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:28.836611 6032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:28.836679 6030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:28.837049 6024 server_base.cc:1034] running on GCE node
I20260927 21:07:28.837271 6024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:28.837502 6024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:28.838662 6024 hybrid_clock.cc:648] HybridClock initialized: now 1790543248838650 us; error 25 us; skew 500 ppm
Sep 27 21:07:28 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:28.841295 6024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:28.841742 6024 webserver.cc:460] Webserver started at http://127.0.98.1:42797/ using document root <none> and password file <none>
I20260927 21:07:28.841914 6024 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:28.841953 6024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:28.842041 6024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:28.843003 6024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "41808efdc5d5411c9d33c169c858434f"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "dc2dcb3a656c5e0d130901d9f8b48173"
server_key_iv: "120e45c4e0c70c6fc98191eee2c6dbf3"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.843382 6024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "41808efdc5d5411c9d33c169c858434f"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "dc2dcb3a656c5e0d130901d9f8b48173"
server_key_iv: "120e45c4e0c70c6fc98191eee2c6dbf3"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.844597 6024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:28.845218 6039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:28.845366 6024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:28.845444 6024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "41808efdc5d5411c9d33c169c858434f"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "dc2dcb3a656c5e0d130901d9f8b48173"
server_key_iv: "120e45c4e0c70c6fc98191eee2c6dbf3"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.845495 6024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:28.855474 6024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:28.857088 6024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:28.857200 6024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:28.857370 6024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:28.857595 6024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:28.857625 6024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:28.857646 6024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:28.857659 6024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:28.863227 6024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41361
I20260927 21:07:28.863320 6152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41361 every 8 connection(s)
I20260927 21:07:28.863633 6024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:07:28 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:28.870694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6024
I20260927 21:07:28.870813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:28.871021 392 external_mini_cluster.cc:1428] Setting key f607e1104f46742739232bf3d29eab59
I20260927 21:07:28.872597 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:28.864366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36773 (local address 127.0.98.62:43295)
0927 21:07:28.864446 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:28.864448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:28.864615 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:07:28.865510 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:28.865511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:28.865520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:28.865571 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:28.866117 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:28.866589 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:28.867609 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:28.867801 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:28.867852 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:28.869711 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:28.869712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:28.869713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:28.869720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:28.871208 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:28.871642 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:28.871643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:28.871643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:28.871705 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:28.871983 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:28.871983 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:28.871983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:28.872077 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:28.872127 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:28.872474 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 21:07:28.872537 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:07:28.872898 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:28.864329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43295 (local address 127.0.98.1:36773)
0927 21:07:28.864501 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:28.864536 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:28.865427 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:28.865622 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:28.865625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:28.865867 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:28.866046 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:28.866049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:28.866668 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:28.866669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:28.867541 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:28.867543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:28.867867 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:28.867867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:28.867957 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:28.868356 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:28.868364 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:28.869619 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:28.871278 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:28.871285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:28.871287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:28.871562 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:28.871754 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:28.871755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:28.871755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:28.871899 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:28.872142 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:28.872154 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:28.872321 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:07:28.872447 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 21:07:28.872616 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260927 21:07:28.873389 6153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43295
I20260927 21:07:28.873473 6153 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:28.873669 6153 heartbeater.cc:511] Master 127.0.98.62:43295 requested a full tablet report, sending...
I20260927 21:07:28.874187 5962 ts_manager.cc:194] Registered new tserver with Master: 41808efdc5d5411c9d33c169c858434f (127.0.98.1:41361)
I20260927 21:07:28.875097 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36773
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:28.894383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:28.972851 6158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:28.973065 6158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:28.973104 6158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:28.974248 6158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:28.974326 6158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:28.974368 6158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:28.975910 6158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:28.976163 6158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:28.976382 6158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:28.981061 6166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:28.981078 6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:28.981256 6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:28.981321 6158 server_base.cc:1034] running on GCE node
I20260927 21:07:28.981526 6158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:28.981722 6158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:28.982928 6158 hybrid_clock.cc:648] HybridClock initialized: now 1790543248982926 us; error 50 us; skew 500 ppm
Sep 27 21:07:28 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:28.986303 6158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:28.986794 6158 webserver.cc:460] Webserver started at http://127.0.98.2:33131/ using document root <none> and password file <none>
I20260927 21:07:28.987058 6158 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:28.987154 6158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:28.987288 6158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:28.988309 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "de13600c10c54293bfe212d22e13e0c4"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "3174c7b17f906602ddecc439b1f54314"
server_key_iv: "bce4aa2f761e3d84a1caa4ff992df8ff"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.988644 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de13600c10c54293bfe212d22e13e0c4"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "3174c7b17f906602ddecc439b1f54314"
server_key_iv: "bce4aa2f761e3d84a1caa4ff992df8ff"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.989897 6158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:28.990577 6173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:28.990761 6158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:28.990856 6158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "de13600c10c54293bfe212d22e13e0c4"
format_stamp: "Formatted at 2026-09-27 21:07:28 on dist-test-slave-8bgs"
server_key: "3174c7b17f906602ddecc439b1f54314"
server_key_iv: "bce4aa2f761e3d84a1caa4ff992df8ff"
server_key_version: "encryptionkey@0"
I20260927 21:07:28.990942 6158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:29.004662 6158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:29.006839 6158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:29.006994 6158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:29.007236 6158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:29.007517 6158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:29.007571 6158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:29.007609 6158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:29.007637 6158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:29.013777 6158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41289
I20260927 21:07:29.013826 6286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41289 every 8 connection(s)
I20260927 21:07:29.014132 6158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:29.018577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6158
I20260927 21:07:29.018698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:29.018883 392 external_mini_cluster.cc:1428] Setting key 1b5eed9b55ba4c28f7c6ee139bdf693e
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:29.023968 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.014824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43295 (local address 127.0.98.2:45943)
0927 21:07:29.014970 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:29.015008 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:29.015961 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:29.016177 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:29.016181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:29.016401 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:29.016571 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:29.016574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.017183 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.017183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:29.017945 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:29.017947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.018257 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.018258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:29.018348 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.018861 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:29.018869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:29.020826 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:29.022623 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:29.022628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:29.022630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:29.022850 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:29.023058 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:29.023059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:29.023059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:29.023200 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:29.023519 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:29.023523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:29.023635 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 21:07:29.023713 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:07:29.023823 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260927 21:07:29.023976 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.014877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45943 (local address 127.0.98.62:43295)
0927 21:07:29.014924 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.014926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.015107 (+ 181us) server_negotiation.cc:407] Connection header received
0927 21:07:29.016051 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.016052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.016060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.016124 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.016645 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.017091 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.018019 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.018192 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.018239 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.020911 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:29.020912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:29.020913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:29.020919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:29.022542 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.022924 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.022925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.022925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.022991 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.023277 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.023278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.023279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.023439 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:29.023519 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:29.023777 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 21:07:29.023872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:07:29.024487 6287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43295
I20260927 21:07:29.024593 6287 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:29.024771 6287 heartbeater.cc:511] Master 127.0.98.62:43295 requested a full tablet report, sending...
I20260927 21:07:29.025131 5962 ts_manager.cc:194] Registered new tserver with Master: de13600c10c54293bfe212d22e13e0c4 (127.0.98.2:41289)
I20260927 21:07:29.025496 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45943
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:29.043272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:29.122323 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:29.122578 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:29.122602 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:29.123765 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:29.123804 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:29.123831 6292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:29.125221 6292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:29.125391 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:29.125558 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:29.130350 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:29.130522 6292 server_base.cc:1034] running on GCE node
W20260927 21:07:29.130355 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:29.130368 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:29.130862 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:29.131143 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:29.132292 6292 hybrid_clock.cc:648] HybridClock initialized: now 1790543249132275 us; error 27 us; skew 500 ppm
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.134567 6292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:29.135038 6292 webserver.cc:460] Webserver started at http://127.0.98.3:33785/ using document root <none> and password file <none>
I20260927 21:07:29.135289 6292 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:29.135332 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:29.135413 6292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:29.136471 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "d61bea9f43514793bbe11f2e9ce7be3d"
format_stamp: "Formatted at 2026-09-27 21:07:29 on dist-test-slave-8bgs"
server_key: "4ebb166ceaf36a76e59c82f597448548"
server_key_iv: "2ef8f4535a641fa974a27d1de48726e2"
server_key_version: "encryptionkey@0"
I20260927 21:07:29.136766 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d61bea9f43514793bbe11f2e9ce7be3d"
format_stamp: "Formatted at 2026-09-27 21:07:29 on dist-test-slave-8bgs"
server_key: "4ebb166ceaf36a76e59c82f597448548"
server_key_iv: "2ef8f4535a641fa974a27d1de48726e2"
server_key_version: "encryptionkey@0"
I20260927 21:07:29.138023 6292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:07:29.138602 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:29.138747 6292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:29.138823 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "d61bea9f43514793bbe11f2e9ce7be3d"
format_stamp: "Formatted at 2026-09-27 21:07:29 on dist-test-slave-8bgs"
server_key: "4ebb166ceaf36a76e59c82f597448548"
server_key_iv: "2ef8f4535a641fa974a27d1de48726e2"
server_key_version: "encryptionkey@0"
I20260927 21:07:29.138875 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:29.156780 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:29.158614 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:29.158730 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:29.158972 6292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:29.159300 6292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:29.159348 6292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:29.159370 6292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:29.159384 6292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:29.165220 6292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35509
I20260927 21:07:29.165310 6420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35509 every 8 connection(s)
I20260927 21:07:29.165609 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:29.167546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6292
I20260927 21:07:29.167673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:29.167860 392 external_mini_cluster.cc:1428] Setting key 64913c46c0d9405ccfb6a8dfbd6eaf62
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:29.174734 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.166301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58499 (local address 127.0.98.62:43295)
0927 21:07:29.166344 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.166352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.166558 (+ 206us) server_negotiation.cc:407] Connection header received
0927 21:07:29.167697 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.167699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.167712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.167773 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.168602 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.169144 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.170129 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.170318 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.170404 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.172461 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:29.172462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:29.172462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:29.172469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:29.173664 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.174004 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.174005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.174005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.174053 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.174256 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.174257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.174257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.174349 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:29.174396 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:29.174615 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:07:29.174672 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:29.174925 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.166279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43295 (local address 127.0.98.3:58499)
0927 21:07:29.166434 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:29.166474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:29.167561 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:29.167856 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:29.167859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:29.168222 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:29.168498 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:29.168504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.169239 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.169240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:29.170023 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:29.170025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.170426 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.170427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:29.170566 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.171004 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:29.171012 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:29.172384 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:29.173733 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:29.173738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:29.173739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:29.173942 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:29.174107 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:29.174107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:29.174108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:29.174205 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:29.174400 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:29.174403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:29.174509 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 21:07:29.174587 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:07:29.174696 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260927 21:07:29.175434 6421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43295
I20260927 21:07:29.175508 6421 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:29.175684 6421 heartbeater.cc:511] Master 127.0.98.62:43295 requested a full tablet report, sending...
I20260927 21:07:29.176038 5962 ts_manager.cc:194] Registered new tserver with Master: d61bea9f43514793bbe11f2e9ce7be3d (127.0.98.3:35509)
I20260927 21:07:29.176395 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58499
I20260927 21:07:29.179664 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/test-admin.keytab.
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.193261 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 21:07:29.194794 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:29.194069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.0.98.1:41361)
0927 21:07:29.194222 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.194227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.194245 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:07:29.194269 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.194270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.194283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.194353 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.194703 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33788: BlockingRecv error: recv got EOF from 127.0.0.1:33788 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260927 21:07:29.194967 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.195925 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.196755 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:29.204335 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.197240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.98.1:41361)
0927 21:07:29.197289 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.197290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.197382 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:07:29.197456 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.197457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.197467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.197545 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.197940 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.198670 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.199469 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.199789 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.199868 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.201776 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:29.201785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:29.201785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:29.201792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:29.203184 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.203544 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.203547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.203547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.203605 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.203799 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.203800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.203800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.203929 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:29.204078 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:29.204154 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:07:29.204232 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 21:07:29.206020 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:29.205482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.0.98.1:41361)
0927 21:07:29.205523 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.205526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.205563 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:07:29.205632 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.205633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.205643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.205686 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.205977 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33810: BlockingRecv error: recv got EOF from 127.0.0.1:33810 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:07:29.206120 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc)
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.206985 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.207829 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:29.214301 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.208291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.0.98.1:41361)
0927 21:07:29.208346 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.208348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.208381 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:07:29.208454 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.208455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.208462 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.208505 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.208859 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.209331 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.210002 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.210224 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.210299 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.212154 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:29.212155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:29.212155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:29.212162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:29.213264 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.213593 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.213594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.213595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.213650 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.213843 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.213844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.213845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.213936 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:29.214034 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:29.214133 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:07:29.214209 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 21:07:29.215667 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:29.215132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.0.98.1:41361)
0927 21:07:29.215168 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.215170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.215216 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:07:29.215282 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.215283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.215291 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.215343 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.215611 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33824: BlockingRecv error: recv got EOF from 127.0.0.1:33824 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:07:29.215751 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790543220945705-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.216601 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:29.217406 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 21:07:29.223330 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:29.217822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.0.98.1:41361)
0927 21:07:29.217859 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:29.217865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:29.217891 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:07:29.217937 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:29.217938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:29.217944 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:29.217987 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:29.218314 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.218712 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.219392 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:29.219560 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:29.219609 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:29.221381 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:29.221382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:29.221382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:29.221388 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:29.222446 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.222723 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.222723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.222723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.222767 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:29.222948 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:29.222949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:29.222949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:29.223022 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:29.223130 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:29.223229 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:07:29.223281 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:29.223901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6024
I20260927 21:07:29.229270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6158
I20260927 21:07:29.233422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6292
I20260927 21:07:29.237442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 5930
2026-09-27T21:07:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1851 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[6441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[6441](info): set up 2 sockets
Sep 27 21:07:29 dist-test-slave-8bgs krb5kdc[6441](info): commencing operation
krb5kdc: starting...
W20260927 21:07:30.267287 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 21:07:30.375722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37147
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:30.453075 6454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:30.453313 6454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:30.453351 6454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:30.454638 6454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:30.454717 6454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:30.454744 6454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:30.454777 6454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:30.454800 6454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:30.456391 6454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:30.456652 6454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:30.456863 6454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:30.461444 6460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.461462 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.461453 6462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:30.461617 6454 server_base.cc:1034] running on GCE node
I20260927 21:07:30.461880 6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:30.462164 6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:30.463317 6454 hybrid_clock.cc:648] HybridClock initialized: now 1790543250463303 us; error 46 us; skew 500 ppm
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:30.465641 6454 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 21:07:30.466095 6454 webserver.cc:460] Webserver started at http://127.0.98.62:40847/ using document root <none> and password file <none>
I20260927 21:07:30.466306 6454 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:30.466372 6454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:30.466524 6454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:30.467568 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "243a548bacc447189e56dc242752e0fc"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "1968fe4a8bdf8fdb174d09b9f1f3f405"
server_key_iv: "cad9aaf3eb254854ff589855d28a5e0e"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.467914 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "243a548bacc447189e56dc242752e0fc"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "1968fe4a8bdf8fdb174d09b9f1f3f405"
server_key_iv: "cad9aaf3eb254854ff589855d28a5e0e"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.469204 6454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:30.469884 6469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.470055 6454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:30.470157 6454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "243a548bacc447189e56dc242752e0fc"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "1968fe4a8bdf8fdb174d09b9f1f3f405"
server_key_iv: "cad9aaf3eb254854ff589855d28a5e0e"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.470228 6454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:30.488430 6454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:30.490167 6454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:30.490324 6454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:30.494393 6454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37147
I20260927 21:07:30.494429 6521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37147 every 8 connection(s)
I20260927 21:07:30.494768 6454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:30.495760 6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:30.497946 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Bootstrap starting.
I20260927 21:07:30.498508 6522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:30.498796 6522 log.cc:824] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:30.499496 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: No bootstrap required, opened a new log
I20260927 21:07:30.499513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6454
I20260927 21:07:30.499624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:30.499820 392 external_mini_cluster.cc:1428] Setting key 3342d460a1f5a5f13d672393dbd9de2f
I20260927 21:07:30.502071 6522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } }
I20260927 21:07:30.502235 6522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:30.502271 6522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243a548bacc447189e56dc242752e0fc, State: Initialized, Role: FOLLOWER
I20260927 21:07:30.502389 6522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } }
I20260927 21:07:30.502465 6522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:30.502486 6522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:30.502550 6522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:30.503135 6522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } }
I20260927 21:07:30.503237 6522 leader_election.cc:302] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 243a548bacc447189e56dc242752e0fc; no voters:
I20260927 21:07:30.503372 6522 leader_election.cc:288] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:30.503438 6527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:30.503593 6527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 LEADER]: Becoming Leader. State: Replica: 243a548bacc447189e56dc242752e0fc, State: Running, Role: LEADER
I20260927 21:07:30.503649 6522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:30.503675 6527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } }
I20260927 21:07:30.504081 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243a548bacc447189e56dc242752e0fc. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } }
I20260927 21:07:30.504087 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } }
I20260927 21:07:30.504244 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:30.504256 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:30.504547 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:30.505062 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:30.506445 6536 catalog_manager.cc:1347] Generated new cluster ID: 2b9575b6379c40a998af10a534adb1f6
I20260927 21:07:30.506497 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:30.511830 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.500294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.0.98.62:37147)
0927 21:07:30.500477 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:30.500489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:30.500520 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:07:30.502028 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:30.502037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:30.502097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:30.502251 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:30.502961 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.504462 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.505651 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.506215 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.506315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.508518 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:30.508522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:30.508522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:30.508540 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:30.510302 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.510682 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.510685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.510685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.510775 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.510980 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.510981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.510981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.511281 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:30.511440 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:30.511541 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:07:30.511673 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260927 21:07:30.515368 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:30.515844 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:30.525153 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Generated new TSK 0
I20260927 21:07:30.525326 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 21:07:30.549176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37147
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:30.627795 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:30.628000 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:30.628036 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:30.629153 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:30.629217 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:30.629261 6548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:30.630654 6548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:30.630901 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:30.631146 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:30.635874 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:30.636013 6548 server_base.cc:1034] running on GCE node
W20260927 21:07:30.635885 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.635877 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:30.636256 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:30.636518 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:30.637681 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790543250637660 us; error 32 us; skew 500 ppm
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:30.640075 6548 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:07:30.640475 6548 webserver.cc:460] Webserver started at http://127.0.98.1:37647/ using document root <none> and password file <none>
I20260927 21:07:30.640712 6548 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:30.640779 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:30.640925 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:30.641919 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ecf43902fdf141bc8c06e65a5cf881d7"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "49cfdb72b00a6b9f370790f79e76bb45"
server_key_iv: "cb289d4f514cf36c34e1868b0b349676"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.642261 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ecf43902fdf141bc8c06e65a5cf881d7"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "49cfdb72b00a6b9f370790f79e76bb45"
server_key_iv: "cb289d4f514cf36c34e1868b0b349676"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.643610 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:07:30.644310 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.644469 6548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:30.644562 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "ecf43902fdf141bc8c06e65a5cf881d7"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "49cfdb72b00a6b9f370790f79e76bb45"
server_key_iv: "cb289d4f514cf36c34e1868b0b349676"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.644649 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:30.664508 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:30.666198 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:30.666370 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:30.666579 6548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:30.666870 6548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:30.666924 6548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.666966 6548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:30.666994 6548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.672465 6548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35131
I20260927 21:07:30.672542 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35131 every 8 connection(s)
I20260927 21:07:30.672906 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:30.673791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6548
I20260927 21:07:30.673898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:30.674118 392 external_mini_cluster.cc:1428] Setting key 63e5f1589a2041b51d2dbaddb45c916f
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:07:30.683212 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.673637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53775 (local address 127.0.98.62:37147)
0927 21:07:30.673679 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:30.673681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:30.673942 (+ 261us) server_negotiation.cc:407] Connection header received
0927 21:07:30.674586 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:30.674587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:30.674599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:30.674650 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:30.675659 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.676424 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.677796 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.677986 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.678040 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.680300 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:30.680302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:30.680302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:30.680310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:30.681852 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.682224 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.682225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.682226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.682286 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.682507 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.682508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.682508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.682636 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:30.682715 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:30.682987 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:07:30.683072 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:30.683281 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.673635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:53775)
0927 21:07:30.673806 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:30.673865 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:30.674471 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:30.674722 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:30.674726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:30.675373 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:30.675555 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:30.675560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.676523 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.676524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:30.677705 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:30.677708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.678072 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.678073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:30.678223 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.678729 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:30.678740 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:30.680199 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:30.681931 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:30.681937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:30.681939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:30.682145 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:30.682356 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:30.682356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:30.682356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:30.682445 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:30.682731 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:30.682736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:30.682858 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 21:07:30.682942 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:07:30.683050 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260927 21:07:30.683786 6677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37147
I20260927 21:07:30.683862 6677 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:30.684047 6677 heartbeater.cc:511] Master 127.0.98.62:37147 requested a full tablet report, sending...
I20260927 21:07:30.684537 6486 ts_manager.cc:194] Registered new tserver with Master: ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 21:07:30.685293 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 21:07:30.697945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37147
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:30.776651 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:30.776889 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:30.776933 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:30.778080 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:30.778147 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:30.778192 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:30.779678 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:30.779943 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:30.780174 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:30.784950 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.784973 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.784950 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:30.785107 6682 server_base.cc:1034] running on GCE node
I20260927 21:07:30.785403 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:30.785655 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:30.786815 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790543250786801 us; error 31 us; skew 500 ppm
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:30.789291 6682 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:07:30.789798 6682 webserver.cc:460] Webserver started at http://127.0.98.2:36243/ using document root <none> and password file <none>
I20260927 21:07:30.790041 6682 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:30.790110 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:30.790266 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:30.791357 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "10f3f863734b4118ab156dda40c0811a"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "f80de938c475407d1a04aa68529ee224"
server_key_iv: "f1bfc397ed940f838e1acd47c5632a22"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.791707 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "10f3f863734b4118ab156dda40c0811a"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "f80de938c475407d1a04aa68529ee224"
server_key_iv: "f1bfc397ed940f838e1acd47c5632a22"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.792959 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:30.793612 6697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.793772 6682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:30.793866 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "10f3f863734b4118ab156dda40c0811a"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "f80de938c475407d1a04aa68529ee224"
server_key_iv: "f1bfc397ed940f838e1acd47c5632a22"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.793944 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:30.802342 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:30.804411 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:30.804518 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:30.804744 6682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:30.805033 6682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:30.805065 6682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.805117 6682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:30.805146 6682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.810515 6682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40637
I20260927 21:07:30.810598 6810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40637 every 8 connection(s)
I20260927 21:07:30.810891 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:30.812090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6682
I20260927 21:07:30.812237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:30.812417 392 external_mini_cluster.cc:1428] Setting key d227c312ee5f6a57302e804278b4c80e
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:07:30.820453 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.811607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34533 (local address 127.0.98.62:37147)
0927 21:07:30.811691 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:30.811693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:30.811861 (+ 168us) server_negotiation.cc:407] Connection header received
0927 21:07:30.812439 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:30.812440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:30.812451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:30.812514 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:30.813501 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.813977 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.815475 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.815666 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.815750 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.817753 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:30.817755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:30.817755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:30.817762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:30.819132 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.819503 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.819504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.819505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.819565 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.819813 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.819814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.819814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.819922 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:30.819996 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:30.820255 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:07:30.820339 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:07:30.820660 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.811570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:34533)
0927 21:07:30.811741 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:30.811776 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:30.812330 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:30.812575 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:30.812579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:30.813238 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:30.813406 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:30.813410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.814068 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.814069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:30.815362 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:30.815365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.815759 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.815760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:30.815890 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.816372 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:30.816388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:30.817664 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:30.819196 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:30.819202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:30.819204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:30.819415 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:30.819635 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:30.819636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:30.819637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:30.819746 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:30.819989 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:30.819994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:30.820130 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:07:30.820234 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:07:30.820414 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260927 21:07:30.821133 6811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37147
I20260927 21:07:30.821208 6811 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:30.821395 6811 heartbeater.cc:511] Master 127.0.98.62:37147 requested a full tablet report, sending...
I20260927 21:07:30.821774 6486 ts_manager.cc:194] Registered new tserver with Master: 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637)
I20260927 21:07:30.822139 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 21:07:30.834969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37147
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:30.914011 6816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:30.914239 6816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:30.914261 6816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:30.915342 6816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:30.915376 6816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:30.915401 6816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:30.916714 6816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:30.916880 6816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:30.917038 6816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:30.921697 6822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.921697 6824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:30.921757 6821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:30.921892 6816 server_base.cc:1034] running on GCE node
I20260927 21:07:30.922065 6816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:30.922312 6816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:30.923484 6816 hybrid_clock.cc:648] HybridClock initialized: now 1790543250923470 us; error 25 us; skew 500 ppm
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:30.925853 6816 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:07:30.926303 6816 webserver.cc:460] Webserver started at http://127.0.98.3:45413/ using document root <none> and password file <none>
I20260927 21:07:30.926515 6816 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:30.926580 6816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:30.926733 6816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:30.927786 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "479382157b096c5a121b8f8fe93b8321"
server_key_iv: "9467494f032782a84f719a7e2553436f"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.928157 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "479382157b096c5a121b8f8fe93b8321"
server_key_iv: "9467494f032782a84f719a7e2553436f"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.929407 6816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:30.930095 6831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.930262 6816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:30.930362 6816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "479382157b096c5a121b8f8fe93b8321"
server_key_iv: "9467494f032782a84f719a7e2553436f"
server_key_version: "encryptionkey@0"
I20260927 21:07:30.930433 6816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:30.941228 6816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:30.943485 6816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:30.943642 6816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:30.943867 6816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:30.944162 6816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:30.944214 6816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.944248 6816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:30.944275 6816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:30.950057 6816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41875
I20260927 21:07:30.950110 6944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41875 every 8 connection(s)
I20260927 21:07:30.950434 6816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:30 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:30.959146 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.951145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51273 (local address 127.0.98.62:37147)
0927 21:07:30.951229 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:30.951231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:30.951402 (+ 171us) server_negotiation.cc:407] Connection header received
0927 21:07:30.951899 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:30.951901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:30.951913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:30.951975 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:30.952876 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.953384 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.954350 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.954557 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.954632 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.956700 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:30.956702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:30.956702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:30.956709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:30.957788 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.958189 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.958190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.958191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.958252 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:30.958492 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:30.958493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:30.958493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:30.958610 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:30.958686 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:30.958942 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:07:30.959018 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:07:30.959311 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:30.951088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:51273)
0927 21:07:30.951276 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:30.951322 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:30.951798 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:30.952041 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:30.952045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:30.952610 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:30.952780 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:30.952784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.953474 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.953476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:30.954254 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:30.954256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:30.954636 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:30.954637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:30.954763 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:30.955262 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:30.955269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:30.956605 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:30.957883 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:30.957889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:30.957891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:30.958105 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:30.958316 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:30.958317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:30.958317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:30.958424 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:30.958677 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:30.958683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:30.958819 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:07:30.958925 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:07:30.959074 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260927 21:07:30.959405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6816
I20260927 21:07:30.959527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:30.959712 392 external_mini_cluster.cc:1428] Setting key 6db9a83f512346703831a5a5c311a90b
I20260927 21:07:30.959717 6945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37147
I20260927 21:07:30.959818 6945 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:30.959995 6945 heartbeater.cc:511] Master 127.0.98.62:37147 requested a full tablet report, sending...
I20260927 21:07:30.960412 6486 ts_manager.cc:194] Registered new tserver with Master: c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875)
I20260927 21:07:30.960841 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51273
I20260927 21:07:30.971513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 21:07:30.993383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35815
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37147,127.0.98.61:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:31.072666 6950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:31.072866 6950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:31.072901 6950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:31.074076 6950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:31.074137 6950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:31.074163 6950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:31.074204 6950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:31.074229 6950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:31.075695 6950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37147,127.0.98.61:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
W20260927 21:07:31.075812 6950 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37147,127.0.98.61:35815'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 21:07:31.075971 6950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:31.076172 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.080837 6955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.080833 6956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.080873 6958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.081043 6950 server_base.cc:1034] running on GCE node
I20260927 21:07:31.081284 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.081600 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.082759 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790543251082746 us; error 42 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.085183 6950 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 21:07:31.085596 6950 webserver.cc:460] Webserver started at http://127.0.98.61:36569/ using document root <none> and password file <none>
I20260927 21:07:31.085824 6950 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.085891 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.086042 6950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:31.087172 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/instance:
uuid: "3e3ff17a6a4249d3b3640b8dc0435a19"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "8f0a880cc17edde0d509e8d0950726e2"
server_key_iv: "1af65ec7456c3c02b34032447e2b22b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.087574 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal/instance:
uuid: "3e3ff17a6a4249d3b3640b8dc0435a19"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "8f0a880cc17edde0d509e8d0950726e2"
server_key_iv: "1af65ec7456c3c02b34032447e2b22b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.088825 6950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:31.089525 6965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.089700 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:31.089794 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
uuid: "3e3ff17a6a4249d3b3640b8dc0435a19"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "8f0a880cc17edde0d509e8d0950726e2"
server_key_iv: "1af65ec7456c3c02b34032447e2b22b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.089879 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.107261 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.108960 6950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:31.109105 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.112846 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35815
I20260927 21:07:31.112879 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35815 every 8 connection(s)
I20260927 21:07:31.113215 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:07:31.114140 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:31.115649 7018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:07:31.117592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 6950
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.125015 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.116259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.98.62:37147)
0927 21:07:31.116305 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.116306 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.116470 (+ 164us) server_negotiation.cc:407] Connection header received
0927 21:07:31.116869 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.116870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.116879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.116940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.117528 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.118127 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.119563 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.119748 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.119822 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.121860 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.121861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.121862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.121882 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.123554 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.124006 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.124007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.124008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.124066 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.124309 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.124309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.124310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.124461 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.124536 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.124803 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.124899 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:31.125234 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.116216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:41980)
0927 21:07:31.116354 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.116392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.116796 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.117010 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.117013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.117234 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.117407 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.117411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.118256 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.118258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.119468 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.119470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.119825 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.119827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.119946 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.120432 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.120440 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.121770 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.123674 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.123680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.123682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.123909 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.124134 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.124135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.124135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.124241 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.124538 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.124543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.124681 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.124782 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.125009 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260927 21:07:31.125946 7018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:07:31.126412 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.118019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.0.98.61:35815)
0927 21:07:31.118184 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.118188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.118208 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:07:31.118249 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.118251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.118272 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.118344 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.119028 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.119727 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.120613 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.120895 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.120975 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.123573 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.123580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.123581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.123587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.125207 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.125584 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.125587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.125588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.125644 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.125848 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.125849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.125849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.125990 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.126152 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.126220 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.126291 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.132879 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.126204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54288)
0927 21:07:31.126243 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.126251 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.126286 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.126699 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.126699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.126836 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.126885 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.126887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.127570 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.127570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.128047 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.128049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.128372 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.128373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.128422 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.128845 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.128849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.129897 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.132001 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.132001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.132002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.132144 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.132306 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.132306 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.132306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.132350 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.132540 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.132541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.132576 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.132670 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.132757 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 21:07:31.132990 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.126232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.0.98.61:35815)
0927 21:07:31.126345 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.126347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.126353 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:07:31.126390 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.126391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.126401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.126672 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.126959 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.127495 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.128118 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.128316 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.128361 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.129968 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.129970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.129970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.129976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.131933 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.132205 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.132206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.132206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.132254 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.132386 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.132387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.132387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.132463 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.132537 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.132693 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.132805 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260927 21:07:31.134156 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Bootstrap starting.
I20260927 21:07:31.134708 7018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:31.134930 7018 log.cc:824] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:31.135567 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: No bootstrap required, opened a new log
I20260927 21:07:31.136847 7018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } }
I20260927 21:07:31.136962 7018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:31.136984 7018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3ff17a6a4249d3b3640b8dc0435a19, State: Initialized, Role: FOLLOWER
I20260927 21:07:31.137102 7018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } }
I20260927 21:07:31.137344 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } } }
I20260927 21:07:31.137416 7018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:31.137423 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:07:31.139006 7036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:07:31.139062 7036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:07:31.140589 6950 master_runner.cc:428] Detected that this master 3e3ff17a6a4249d3b3640b8dc0435a19 is joining an existing cluster
I20260927 21:07:31.140645 6950 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35815
I20260927 21:07:31.148675 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.141631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.0.98.62:37147)
0927 21:07:31.141674 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.141677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.141793 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:07:31.141898 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.141910 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.141919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.141969 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.142427 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.142898 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.144119 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.144388 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.144422 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.145744 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.145745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.145745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.145751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.147366 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.147842 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.147843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.147843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.147903 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.148099 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.148101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.148101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.148253 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.148320 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.148504 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.148583 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:31.149003 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.141593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:41986)
0927 21:07:31.141727 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.141738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.141817 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.142010 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.142011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.142235 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.142334 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.142336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.142998 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.142999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.144015 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.144017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.144502 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.144503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.144586 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.145070 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.145074 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.145640 (+ 566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.147469 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.147470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.147471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.147770 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.147956 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.147957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.147957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.148012 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.148347 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.148348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.148388 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.148775 (+ 387us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.148874 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":13,"thread_start_us":74,"threads_started":1}
I20260927 21:07:31.149601 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.141654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54300)
0927 21:07:31.141773 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.141781 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.141821 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.141979 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.141980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.142232 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.142334 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.142336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.143253 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.143254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.144083 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.144085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.144590 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.144590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.144653 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.145253 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.145258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.145672 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.148052 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.148053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.148053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.148218 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.148416 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.148416 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.148417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.148472 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.148715 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.148716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.148970 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.149375 (+ 405us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.149483 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260927 21:07:31.149729 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.141806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.0.98.61:35815)
0927 21:07:31.141838 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.141840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.141847 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:31.141866 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.141866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.141875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.141959 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.142414 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.143169 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.144170 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.144491 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.144580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.145763 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.145764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.145765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.145771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.147974 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.148297 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.148298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.148298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.148343 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.148538 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.148539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.148539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.148651 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.148724 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.149411 (+ 687us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.149535 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":16}
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.158764 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.151630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.0.98.61:35815)
0927 21:07:31.151674 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.151676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.151788 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:07:31.151854 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.151855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.151864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.151943 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.152293 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.153009 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.153773 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.154071 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.154147 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.156158 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.156159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.156160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.156166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.157570 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.157952 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.157953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.157954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.158011 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.158225 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.158226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.158227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.158348 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.158422 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.158595 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.158668 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:07:31.158790 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.151560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54310)
0927 21:07:31.151684 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.151724 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.151798 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.151949 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.151951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.152182 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.152228 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.152230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.153090 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.153091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.153696 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.153697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.154144 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.154144 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.154207 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.154657 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.154662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.156064 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.157670 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.157677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.157679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.157872 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.158079 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.158079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.158080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.158139 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.158414 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.158420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.158471 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.158570 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.158680 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260927 21:07:31.159530 6486 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35815: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35815
}
attrs {
promote: true
}
}
dest_uuid: "243a548bacc447189e56dc242752e0fc"
cas_config_opid_index: -1
I20260927 21:07:31.159843 6486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } }
I20260927 21:07:31.160292 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3e3ff17a6a4249d3b3640b8dc0435a19 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } }
I20260927 21:07:31.160662 6992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:31.161231 7046 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35815
I20260927 21:07:31.161317 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } }
I20260927 21:07:31.161384 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:31.161465 6950 master.cc:537] Master@127.0.98.61:35815 shutting down...
I20260927 21:07:31.161579 6992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Refusing update from remote peer 243a548bacc447189e56dc242752e0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 21:07:31.161906 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:07:31.162758 6471 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 21:07:31.163144 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc -> Peer 3e3ff17a6a4249d3b3640b8dc0435a19 (127.0.98.61:35815): Couldn't send request to peer 3e3ff17a6a4249d3b3640b8dc0435a19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:07:31.164929 6950 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:31.165035 6950 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:31.165086 6950 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: stopping tablet replica
I20260927 21:07:31.177937 6950 master.cc:552] Master@127.0.98.61:35815 shutdown complete.
I20260927 21:07:31.177999 6950 master_runner.cc:305] Clearing existing system tablet
I20260927 21:07:31.178123 6950 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.178184 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.178785 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:31.179337 7050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.179458 6950 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.179512 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.179538 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
uuid: "3e3ff17a6a4249d3b3640b8dc0435a19"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "8f0a880cc17edde0d509e8d0950726e2"
server_key_iv: "1af65ec7456c3c02b34032447e2b22b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.180119 6950 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:31.181449 6950 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:07:31.181517 6950 log.cc:1196] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 21:07:31.181734 6950 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Deleting consensus metadata
I20260927 21:07:31.181855 6950 master_runner.cc:315] Copying system tablet from 127.0.98.62:37147
I20260927 21:07:31.182436 6950 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37147
I20260927 21:07:31.188792 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.182807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.0.98.62:37147)
0927 21:07:31.182851 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.182853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.182910 (+ 57us) server_negotiation.cc:407] Connection header received
0927 21:07:31.182990 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.182991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.182999 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.183042 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.183461 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.183982 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.185036 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.185212 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.185260 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.186250 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.186251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.186251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.186257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.187783 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.188152 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.188153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.188154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.188203 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.188386 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.188386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.188387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.188476 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.188522 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.188677 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.188733 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:07:31.188855 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.182726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:41994)
0927 21:07:31.182840 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.182851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.182927 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.183057 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.183058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.183284 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.183350 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.183352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.184063 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.184064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.184955 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.184958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.185281 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.185282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.185351 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.185768 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.185773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.186166 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.187856 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.187857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.187857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.188078 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.188263 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.188264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.188264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.188327 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.188535 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.188535 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.188581 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.188651 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.188747 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20260927 21:07:31.189121 6506 tablet_copy_service.cc:140] P 243a548bacc447189e56dc242752e0fc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3e3ff17a6a4249d3b3640b8dc0435a19 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41994)
I20260927 21:07:31.189227 6506 tablet_copy_service.cc:161] P 243a548bacc447189e56dc242752e0fc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3e3ff17a6a4249d3b3640b8dc0435a19 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41994: session id = 3e3ff17a6a4249d3b3640b8dc0435a19-00000000000000000000000000000000
I20260927 21:07:31.190029 6506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:07:31.190614 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:07:31.191506 6950 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: tablet copy: Starting download of 0 data blocks...
I20260927 21:07:31.191664 6950 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: tablet copy: Starting download of 1 WAL segments...
I20260927 21:07:31.192549 6950 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:07:31.193274 6506 tablet_copy_service.cc:342] P 243a548bacc447189e56dc242752e0fc: Request end of tablet copy session 3e3ff17a6a4249d3b3640b8dc0435a19-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41994
I20260927 21:07:31.193334 6506 tablet_copy_service.cc:433] P 243a548bacc447189e56dc242752e0fc: ending tablet copy session 3e3ff17a6a4249d3b3640b8dc0435a19-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3e3ff17a6a4249d3b3640b8dc0435a19
I20260927 21:07:31.195719 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.196780 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.196838 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.199957 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.200014 6950 server_base.cc:1034] running on GCE node
I20260927 21:07:31.200169 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.200309 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.201433 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790543251201418 us; error 30 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.202656 6950 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 21:07:31.202917 6950 webserver.cc:460] Webserver started at http://127.0.98.61:39161/ using document root <none> and password file <none>
I20260927 21:07:31.203030 6950 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.203063 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.203691 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.204206 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.204341 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.204423 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
uuid: "3e3ff17a6a4249d3b3640b8dc0435a19"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "8f0a880cc17edde0d509e8d0950726e2"
server_key_iv: "1af65ec7456c3c02b34032447e2b22b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.204480 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.217221 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.218801 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.221799 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35815
I20260927 21:07:31.221817 7119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35815 every 8 connection(s)
I20260927 21:07:31.222096 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:07:31.222929 7120 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:07:31.223503 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Bootstrap starting.
I20260927 21:07:31.226341 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:07:31.226560 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Bootstrap complete.
I20260927 21:07:31.226747 7120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } }
I20260927 21:07:31.226836 7120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e3ff17a6a4249d3b3640b8dc0435a19, State: Initialized, Role: LEARNER
I20260927 21:07:31.226874 7120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } }
I20260927 21:07:31.227056 7120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:31.227056 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } }
I20260927 21:07:31.227227 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:07:31.228679 7135 catalog_manager.cc:1259] Loaded cluster ID: 2b9575b6379c40a998af10a534adb1f6
I20260927 21:07:31.229001 7135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: loading cluster ID for follower catalog manager: success
I20260927 21:07:31.230348 7135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: acquiring CA information for follower catalog manager: success
I20260927 21:07:31.230648 7135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:07:31.235780 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.227961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.0.98.61:35815)
0927 21:07:31.228064 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.228066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.228073 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:31.228095 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.228096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.228107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.228882 (+ 775us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.228933 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.230443 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.231538 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.231731 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.231785 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.233123 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.233125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.233125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.233131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.234636 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.235026 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.235026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.235027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.235074 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.235342 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.235343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.235343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.235435 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.235517 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.235657 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.235721 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 21:07:31.259656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41059
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:31.339020 7138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:31.339226 7138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:31.339246 7138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:31.340344 7138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:31.340382 7138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:31.340396 7138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:31.340408 7138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:31.340456 7138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:31.341856 7138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:31.342079 7138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:31.342296 7138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.346848 7146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.347000 7144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.347056 7143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.347203 7138 server_base.cc:1034] running on GCE node
I20260927 21:07:31.347532 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.347805 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.348956 7138 hybrid_clock.cc:648] HybridClock initialized: now 1790543251348950 us; error 35 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.351264 7138 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 21:07:31.351657 7138 webserver.cc:460] Webserver started at http://127.0.98.60:39443/ using document root <none> and password file <none>
I20260927 21:07:31.351873 7138 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.351939 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.352065 7138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:31.353077 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/instance:
uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "3ced37d9abb48466b96be2f22fd88f13"
server_key_iv: "eee0b88bd6a9946fdbc014dbffe5c216"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.353436 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal/instance:
uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "3ced37d9abb48466b96be2f22fd88f13"
server_key_iv: "eee0b88bd6a9946fdbc014dbffe5c216"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.354656 7138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:31.355351 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.355504 7138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:31.355594 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "3ced37d9abb48466b96be2f22fd88f13"
server_key_iv: "eee0b88bd6a9946fdbc014dbffe5c216"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.355677 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.382696 7138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.384656 7138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:31.384815 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.389038 7138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41059
I20260927 21:07:31.389066 7205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41059 every 8 connection(s)
I20260927 21:07:31.389427 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:07:31.390378 7206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:31.391973 7206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:07:31.394081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 7138
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:07:31.401254 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.392718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.0.98.62:37147)
0927 21:07:31.392771 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.392773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.392863 (+ 90us) server_negotiation.cc:407] Connection header received
0927 21:07:31.393284 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.393285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.393294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.393350 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.393955 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.394678 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.395789 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.396091 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.396162 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.398379 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.398381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.398381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.398388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.399928 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.400312 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.400313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.400314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.400373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.400626 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.400627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.400627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.400738 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.400817 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.401088 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.401163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:07:31.401446 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.392583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42004)
0927 21:07:31.392727 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.392765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.393208 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.393420 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.393423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.393660 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.393844 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.393848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.394805 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.394806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.395703 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.395706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.396171 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.396172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.396259 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.396758 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.396766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.398302 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.400009 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.400014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.400016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.400232 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.400449 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.400449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.400450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.400553 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.400807 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.400831 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.400960 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.401068 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.401218 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":47,"thread_start_us":81,"threads_started":1}
I20260927 21:07:31.401928 7206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:07:31.403615 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.394591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.0.98.60:41059)
0927 21:07:31.394797 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.394802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.394829 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:07:31.394847 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.394849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.394863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.394950 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.395554 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.396527 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.397206 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.397656 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.397723 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.400372 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.400379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.400380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.400386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.402322 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.402655 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.402658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.402658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.402712 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.402914 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.402915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.402916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.403071 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.403309 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.403379 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.403494 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":177,"thread_start_us":132,"threads_started":1}
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.408366 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.402183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54340)
0927 21:07:31.402222 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.402230 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.402261 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.402447 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.402448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.402578 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.402607 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.402608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.403434 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.403435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.404093 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.404095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.404574 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.404575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.404631 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.404964 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.404968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.406044 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.407633 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.407634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.407634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.407785 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.407962 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.407963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.407963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.408006 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.408176 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.408176 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.408206 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.408253 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.408313 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 21:07:31.408433 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.402264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.0.98.61:35815)
0927 21:07:31.402330 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.402332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.402339 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:31.402364 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.402365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.402374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.402446 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.402689 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.403323 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.404217 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.404512 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.404566 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.406118 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.406120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.406120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.406125 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.407551 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.407843 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.407844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.407844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.407892 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.408042 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.408043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.408043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.408127 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.408174 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.408268 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.408331 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:07:31.408893 7206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:07:31.414106 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.409132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.0.1:50952)
0927 21:07:31.409184 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.409195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.409232 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.409339 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.409339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.409439 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.409463 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.409464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.410024 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.410024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.410443 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.410444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.410760 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.410760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.410793 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.411182 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.411187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.412168 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.413369 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.413370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.413370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.413519 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.413678 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.413679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.413679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.413725 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.413890 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.413891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.413921 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.413969 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.414035 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:07:31.414166 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.409189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.0.98.60:41059)
0927 21:07:31.409221 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.409223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.409229 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:07:31.409289 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.409289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.409296 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.409336 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.409514 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.409965 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.410502 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.410681 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.410744 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.412239 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.412240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.412241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.412247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.413304 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.413581 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.413581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.413581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.413623 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.413769 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.413769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.413770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.413845 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.413886 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.413983 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.414075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":18}
I20260927 21:07:31.415455 7206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Bootstrap starting.
I20260927 21:07:31.416254 7206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:31.416531 7206 log.cc:824] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:31.417224 7206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: No bootstrap required, opened a new log
I20260927 21:07:31.418994 7206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } }
I20260927 21:07:31.419216 7206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:31.419271 7206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f44e091d9d4c5a87fa0d21c2343e25, State: Initialized, Role: FOLLOWER
I20260927 21:07:31.419397 7206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } }
I20260927 21:07:31.419691 7206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:31.419661 7211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } } }
I20260927 21:07:31.419766 7211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:07:31.421277 7222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:07:31.421334 7222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:07:31.423568 7138 master_runner.cc:428] Detected that this master 60f44e091d9d4c5a87fa0d21c2343e25 is joining an existing cluster
I20260927 21:07:31.423616 7138 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41059
I20260927 21:07:31.432317 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.424654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.0.98.62:37147)
0927 21:07:31.424696 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.424698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.424988 (+ 290us) server_negotiation.cc:407] Connection header received
0927 21:07:31.425087 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.425088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.425099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.425128 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.425680 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.426439 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427632 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.427940 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.428000 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.429393 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.429393 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.429394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.429400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.430856 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.431643 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.431644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.431644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.431699 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.431892 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.431893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.431893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.431984 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.432032 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.432205 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.432259 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:31.432374 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.424744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42008)
0927 21:07:31.424923 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.424933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.425014 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.425206 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.425207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.425528 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.425586 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.425588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.426537 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.426537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.427496 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.427499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.428433 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.428435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.428499 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.428869 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.428874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.429302 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.430937 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.430938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.430939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.431561 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.431762 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.431763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.431763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.431828 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.432044 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.432045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.432098 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.432187 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.432279 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":54,"threads_started":1}
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:07:31.450749 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.434359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.0.1:50972)
0927 21:07:31.434404 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.434414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.434451 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.434676 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.434677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.434820 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.434852 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.434853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.435627 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.435628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.436113 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.436114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.436511 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.436512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.436576 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.436942 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.436946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.438245 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.449631 (+ 11386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.449632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.449633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.449857 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.450076 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.450078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.450149 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.450429 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.450430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.450477 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.450528 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.450640 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 21:07:31.450941 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.424859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.98.60:41059)
0927 21:07:31.424891 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.424893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.424978 (+ 85us) server_negotiation.cc:407] Connection header received
0927 21:07:31.425070 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.425070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.425079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.425121 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.425642 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.426352 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427421 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.427709 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427745 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.447607 (+ 19862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.447608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.447609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.447615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.449904 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450258 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.450258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.450259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.450299 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450462 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.450463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.450463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.450576 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.450613 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.450773 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.450841 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:07:31.451361 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.434422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.98.60:41059)
0927 21:07:31.434562 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.434563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.434568 (+ 5us) server_negotiation.cc:407] Connection header received
0927 21:07:31.434593 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.434594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.434606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.434674 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.434928 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.435540 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.436201 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.436429 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.436497 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.438342 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.438343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.438343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.438350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.449526 (+ 11176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.449941 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.449942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.449943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.450004 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450221 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.450222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.450222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.450352 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.450978 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.451122 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.451215 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260927 21:07:31.451452 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.424815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.0.1:50966)
0927 21:07:31.424929 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.424938 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.425001 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.425176 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.425177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.425514 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.425570 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.425572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.426451 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.426451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.427337 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.427339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.427852 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.427926 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.428397 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.428401 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.447478 (+ 19077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.449999 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.449999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.450000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.450188 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.450358 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.450359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.450403 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.450650 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.450650 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.450681 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.451113 (+ 432us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.451298 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":85,"thread_start_us":68,"threads_started":1}
I20260927 21:07:31.451833 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.424662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.0.98.61:35815)
0927 21:07:31.424701 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.424703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.424981 (+ 278us) server_negotiation.cc:407] Connection header received
0927 21:07:31.425087 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.425088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.425098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.425127 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.425646 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.426353 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427474 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.427773 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427810 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.448468 (+ 20658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.448470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.448470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.448478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.450065 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450408 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.450409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.450409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.450457 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450658 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.450659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.450660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.450777 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.450835 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.451633 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.451720 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:31.451932 6486 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41059: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41059
}
attrs {
promote: true
}
}
dest_uuid: "243a548bacc447189e56dc242752e0fc"
cas_config_opid_index: 5
I20260927 21:07:31.452183 6486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } }
I20260927 21:07:31.452535 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.424605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54356)
0927 21:07:31.424920 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.424929 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.425013 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.425181 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.425182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.425471 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.425543 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.425545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.426439 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.426440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.427393 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.427395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.427872 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.427872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.427949 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.447815 (+ 19866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.447821 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.448359 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.450141 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.450142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.450330 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.450523 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.450524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.450524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.450576 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.451180 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.451181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.451218 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.451284 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.451381 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":62,"thread_start_us":79,"threads_started":1}
I20260927 21:07:31.452510 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.453256 7046 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41059
I20260927 21:07:31.453346 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.453415 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:31.453490 7138 master.cc:537] Master@127.0.98.60:41059 shutting down...
I20260927 21:07:31.453656 7180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:31.454650 7180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Refusing update from remote peer 243a548bacc447189e56dc242752e0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:07:31.455749 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:07:31.456544 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc -> Peer 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60:41059): Couldn't send request to peer 60f44e091d9d4c5a87fa0d21c2343e25. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:07:31.457123 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.452585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54364)
0927 21:07:31.452624 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.452636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.452680 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.453801 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.453802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.453804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:31.453844 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.453845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.454707 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.454708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.455944 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.455947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.456872 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.456873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.456934 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.456952 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.457011 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.457051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:07:31.457127 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.453489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.0.98.61:35815)
0927 21:07:31.453524 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.453526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.453535 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:31.453555 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.453556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.453575 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.453644 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:31.453923 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.454613 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.456110 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.456703 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.456764 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.456843 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.457029 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.457068 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 21:07:31.457362 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 LEARNER]: Refusing update from remote peer 243a548bacc447189e56dc242752e0fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:07:31.457562 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 21:07:31.458105 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.458633 7138 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:31.458735 7138 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:31.458791 7138 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: stopping tablet replica
I20260927 21:07:31.459401 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243a548bacc447189e56dc242752e0fc. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.459519 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:07:31.459571 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: true } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.459615 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:07:31.460525 7045 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: attempting to promote NON_VOTER 3e3ff17a6a4249d3b3640b8dc0435a19 to VOTER
I20260927 21:07:31.460711 7045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } }
W20260927 21:07:31.461221 6439 connection.cc:535] client connection to 127.0.98.60:41059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:07:31.461401 6471 connection.cc:535] client connection to 127.0.98.60:41059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:07:31.461414 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 LEARNER]: Refusing update from remote peer 243a548bacc447189e56dc242752e0fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:07:31.461714 7234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 21:07:31.461774 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.461621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.0.1:50988)
0927 21:07:31.461664 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.461724 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:07:31.461865 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc -> Peer 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60:41059): Couldn't send request to peer 60f44e091d9d4c5a87fa0d21c2343e25. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:07:31.462291 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3e3ff17a6a4249d3b3640b8dc0435a19 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.462994 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.463079 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:31.463053 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3e3ff17a6a4249d3b3640b8dc0435a19 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.462994 7046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.463234 7046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:31.463820 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.463925 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:07:31.472457 7138 master.cc:552] Master@127.0.98.60:41059 shutdown complete.
I20260927 21:07:31.472520 7138 master_runner.cc:305] Clearing existing system tablet
I20260927 21:07:31.472662 7138 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.472724 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.473420 7138 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:31.473904 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.474013 7138 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.474068 7138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.474090 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "3ced37d9abb48466b96be2f22fd88f13"
server_key_iv: "eee0b88bd6a9946fdbc014dbffe5c216"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.474702 7138 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:31.476118 7138 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:07:31.476174 7138 log.cc:1196] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 21:07:31.476377 7138 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Deleting consensus metadata
I20260927 21:07:31.476500 7138 master_runner.cc:315] Copying system tablet from 127.0.98.62:37147
I20260927 21:07:31.477124 7138 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37147
I20260927 21:07:31.484153 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.477421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42024)
0927 21:07:31.477628 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.477638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.477694 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.477828 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.477829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.477975 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.478033 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.478035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.478958 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.478959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.479563 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.479566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.480034 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.480035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.480083 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.481265 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.481269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.481552 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.483276 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.483277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.483278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.483452 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.483642 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.483643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.483643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.483691 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.483935 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.483936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.483969 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.484018 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.484088 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260927 21:07:31.484201 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.477487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.0.98.62:37147)
0927 21:07:31.477537 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.477539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.477700 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:07:31.477735 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.477736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.477746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.477820 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.478098 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.478877 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.479639 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.479956 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.480023 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.481640 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.481641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.481641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.481649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.483193 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.483517 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.483518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.483519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.483579 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.483747 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.483748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.483748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.483872 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.483946 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.484042 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.484113 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:07:31.484359 6506 tablet_copy_service.cc:140] P 243a548bacc447189e56dc242752e0fc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60f44e091d9d4c5a87fa0d21c2343e25 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42024)
I20260927 21:07:31.484397 6506 tablet_copy_service.cc:161] P 243a548bacc447189e56dc242752e0fc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60f44e091d9d4c5a87fa0d21c2343e25 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42024: session id = 60f44e091d9d4c5a87fa0d21c2343e25-00000000000000000000000000000000
I20260927 21:07:31.484951 6506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:07:31.485383 7138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:07:31.486266 7138 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: tablet copy: Starting download of 0 data blocks...
I20260927 21:07:31.486428 7138 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: tablet copy: Starting download of 1 WAL segments...
I20260927 21:07:31.487149 7138 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:07:31.487996 6506 tablet_copy_service.cc:342] P 243a548bacc447189e56dc242752e0fc: Request end of tablet copy session 60f44e091d9d4c5a87fa0d21c2343e25-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42024
I20260927 21:07:31.488062 6506 tablet_copy_service.cc:433] P 243a548bacc447189e56dc242752e0fc: ending tablet copy session 60f44e091d9d4c5a87fa0d21c2343e25-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60f44e091d9d4c5a87fa0d21c2343e25
I20260927 21:07:31.490340 7138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.491482 7243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.491714 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.491744 7246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.491892 7138 server_base.cc:1034] running on GCE node
I20260927 21:07:31.491977 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.492116 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.493240 7138 hybrid_clock.cc:648] HybridClock initialized: now 1790543251493232 us; error 23 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.494539 7138 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 21:07:31.494807 7138 webserver.cc:460] Webserver started at http://127.0.98.60:35339/ using document root <none> and password file <none>
I20260927 21:07:31.494916 7138 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.494944 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.495617 7138 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.496075 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.496165 7138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.496194 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
format_stamp: "Formatted at 2026-09-27 21:07:31 on dist-test-slave-8bgs"
server_key: "3ced37d9abb48466b96be2f22fd88f13"
server_key_iv: "eee0b88bd6a9946fdbc014dbffe5c216"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.496238 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.534565 7138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.536242 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.539764 7138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41059
I20260927 21:07:31.539896 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41059 every 8 connection(s)
I20260927 21:07:31.540112 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:07:31.540807 7306 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:07:31.541216 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Bootstrap starting.
I20260927 21:07:31.543483 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:07:31.543721 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: Bootstrap complete.
I20260927 21:07:31.543910 7306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } }
I20260927 21:07:31.543995 7306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60f44e091d9d4c5a87fa0d21c2343e25, State: Initialized, Role: LEARNER
I20260927 21:07:31.544046 7306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } }
I20260927 21:07:31.544222 7306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:31.544216 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:31.544297 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:07:31.545606 7320 catalog_manager.cc:1259] Loaded cluster ID: 2b9575b6379c40a998af10a534adb1f6
I20260927 21:07:31.545673 7320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: loading cluster ID for follower catalog manager: success
I20260927 21:07:31.546638 7320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: acquiring CA information for follower catalog manager: success
I20260927 21:07:31.546912 7320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:07:31.568668 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.562187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.0.98.60:41059)
0927 21:07:31.562302 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.562304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.562314 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:31.562338 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.562339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.562350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.562421 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.562890 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.563975 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.564634 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.564936 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.565010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.566121 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.566123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.566123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.566130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.567692 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.567997 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.567998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.567998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.568062 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.568251 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.568252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.568252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.568365 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.568441 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.568515 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.568574 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260927 21:07:31.569224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6548
I20260927 21:07:31.575567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35131
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37147
--webserver_port=37647
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:31.654353 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:31.654523 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:31.654544 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:31.655638 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:31.655676 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:31.655701 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:31.657091 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37647
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:31.657317 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:31.657543 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.662168 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.662169 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.662196 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.662310 7322 server_base.cc:1034] running on GCE node
I20260927 21:07:31.662580 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.662817 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.663950 7322 hybrid_clock.cc:648] HybridClock initialized: now 1790543251663937 us; error 28 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.666334 7322 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:07:31.666735 7322 webserver.cc:460] Webserver started at http://127.0.98.1:37647/ using document root <none> and password file <none>
I20260927 21:07:31.666952 7322 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.667004 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.668287 7322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:31.668864 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.669023 7322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:31.669123 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "ecf43902fdf141bc8c06e65a5cf881d7"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "49cfdb72b00a6b9f370790f79e76bb45"
server_key_iv: "cb289d4f514cf36c34e1868b0b349676"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.669409 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.677150 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.679301 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:31.679454 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.679656 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:31.679920 7322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:31.679970 7322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.680001 7322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:31.680027 7322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.685429 7322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35131
I20260927 21:07:31.685498 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35131 every 8 connection(s)
I20260927 21:07:31.685806 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:31.689755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 7322
I20260927 21:07:31.689903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6682
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.697371 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.686476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40039 (local address 127.0.98.62:37147)
0927 21:07:31.686517 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.686520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.686744 (+ 224us) server_negotiation.cc:407] Connection header received
0927 21:07:31.687436 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.687438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.687544 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.687577 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.688243 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.688837 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690354 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.690661 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690704 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.693840 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.693841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.693841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.693848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.695784 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696260 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.696261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.696262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.696318 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696591 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.696592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.696592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.696738 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.696780 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.697162 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.697244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:31.697455 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.686795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58181 (local address 127.0.98.60:41059)
0927 21:07:31.686834 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.686836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.686927 (+ 91us) server_negotiation.cc:407] Connection header received
0927 21:07:31.687438 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.687439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.687453 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.687487 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.688269 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.689035 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690355 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.690662 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690706 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.694358 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.694359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.694360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.694368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.696271 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696662 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.696663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.696664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.696716 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696914 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.696915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.696915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.697036 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.697075 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.697274 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.697354 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:31.697575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40637
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37147
--webserver_port=36243
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:07:31.697963 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.686735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.98.1:58181)
0927 21:07:31.686868 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.686878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.687314 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.687972 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.687973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.688110 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.688170 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.688171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.689122 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.689123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.690252 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.690254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.690795 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.690936 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.691684 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.691688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.694258 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.696334 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696334 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.696335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.696532 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.696790 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.696791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.696846 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.697120 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.697121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.697162 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.697568 (+ 406us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.697744 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":39,"thread_start_us":58,"threads_started":1}
I20260927 21:07:31.698179 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.686518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53757 (local address 127.0.98.61:35815)
0927 21:07:31.686563 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.686566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.686924 (+ 358us) server_negotiation.cc:407] Connection header received
0927 21:07:31.687415 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.687416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.687427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.687481 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.688230 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.689051 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690348 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.690688 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690729 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.694358 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.694360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.694360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.694367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.696998 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.697378 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.697379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.697379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.697428 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.697619 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.697620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.697620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.697750 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.697791 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.697973 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.698050 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:31.698391 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41059
I20260927 21:07:31.698483 7451 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.698678 7451 heartbeater.cc:511] Master 127.0.98.60:41059 requested a full tablet report, sending...
I20260927 21:07:31.698769 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.686479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:40039)
0927 21:07:31.686623 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.686673 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.687312 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.687657 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.687659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.687950 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.688134 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.688141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.688993 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.688993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.690251 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.690254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.690793 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.690936 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.691599 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.691607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.693740 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.695875 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.695880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.695882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.696166 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.696392 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.696393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.696393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.696520 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.696825 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.696830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.697015 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.698395 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.698574 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":110,"spinlock_wait_cycles":7040,"thread_start_us":95,"threads_started":1}
I20260927 21:07:31.698678 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.686479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.98.1:53757)
0927 21:07:31.686834 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.686844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.687324 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.687645 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.687649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.687940 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.688134 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.688138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.689135 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.689136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.690252 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.690255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.690802 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.690803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.690937 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.691803 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.691808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.694258 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.697075 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.697076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.697076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.697303 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.697493 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.697493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.697494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.697550 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.697837 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.697838 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.697873 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.698301 (+ 428us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.698396 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":80,"threads_started":1}
I20260927 21:07:31.699254 7270 ts_manager.cc:194] Registered new tserver with Master: ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131)
I20260927 21:07:31.699472 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37147
I20260927 21:07:31.699525 7452 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.699613 7452 heartbeater.cc:511] Master 127.0.98.62:37147 requested a full tablet report, sending...
I20260927 21:07:31.699677 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35815
I20260927 21:07:31.699712 7453 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.699764 7453 heartbeater.cc:511] Master 127.0.98.61:35815 requested a full tablet report, sending...
I20260927 21:07:31.699847 6486 ts_manager.cc:194] Re-registered known tserver with Master: ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131)
I20260927 21:07:31.700229 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40039
I20260927 21:07:31.700594 7084 ts_manager.cc:194] Registered new tserver with Master: ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131)
W20260927 21:07:31.776867 7458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:31.777061 7458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:31.777083 7458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:31.778144 7458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:31.778179 7458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:31.778204 7458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:31.779549 7458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36243
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:31.779717 7458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:31.779875 7458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.784581 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.784564 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.784565 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.784592 7458 server_base.cc:1034] running on GCE node
I20260927 21:07:31.784950 7458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.785180 7458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.786332 7458 hybrid_clock.cc:648] HybridClock initialized: now 1790543251786316 us; error 30 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.788805 7458 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:07:31.789247 7458 webserver.cc:460] Webserver started at http://127.0.98.2:36243/ using document root <none> and password file <none>
I20260927 21:07:31.789474 7458 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.789548 7458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.790757 7458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:31.791373 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.791541 7458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:31.791635 7458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "10f3f863734b4118ab156dda40c0811a"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "f80de938c475407d1a04aa68529ee224"
server_key_iv: "f1bfc397ed940f838e1acd47c5632a22"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.791952 7458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.807690 7458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.809356 7458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:31.809505 7458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.809715 7458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:31.809989 7458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:31.810041 7458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.810081 7458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:31.810134 7458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.815958 7458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40637
I20260927 21:07:31.816001 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40637 every 8 connection(s)
I20260927 21:07:31.816308 7458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.822718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 7458
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 21:07:31.822878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6816
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:31.827471 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.817315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51449 (local address 127.0.98.60:41059)
0927 21:07:31.817363 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.817366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.817460 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:07:31.817862 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.817864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.817876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.817908 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.818561 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.819070 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820383 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.820661 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820742 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.823487 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.823488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.823489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.823495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.825277 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826299 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.826300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.826301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.826358 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826664 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.826665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.826666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.826809 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.826852 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.827245 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.827331 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:31.827647 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.816962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34813 (local address 127.0.98.61:35815)
0927 21:07:31.817013 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.817015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.817284 (+ 269us) server_negotiation.cc:407] Connection header received
0927 21:07:31.817864 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.817864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.817876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.817908 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.818614 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.819347 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820225 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.820489 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820550 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.824654 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.824655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.824655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.824664 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.826314 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826714 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.826715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.826716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.826773 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826976 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.826977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.826977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.827148 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.827192 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.827457 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.827543 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:31.828014 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.816962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.98.2:34813)
0927 21:07:31.817144 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.817181 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.817757 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.818356 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.818357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.818498 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.818553 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.818555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.819426 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.819427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.820126 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.820128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.820567 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.820709 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.821175 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.821183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.824557 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.826409 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.826410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.826634 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.826847 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.826848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.826904 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.827267 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.827268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.827309 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.827787 (+ 478us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.827903 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260927 21:07:31.828014 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.817238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.98.2:51449)
0927 21:07:31.817386 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.817394 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.817750 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.818015 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.818016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.818287 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.818451 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.818456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.819226 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.819227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.820305 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.820306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.820740 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.820810 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.821327 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.821332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.823345 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.825359 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.825364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.825366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.826210 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.826426 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.826427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.826588 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.826897 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.826903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.827059 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.827563 (+ 504us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.827742 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260927 21:07:31.828612 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35815
I20260927 21:07:31.828713 7589 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.828765 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.817042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47483 (local address 127.0.98.62:37147)
0927 21:07:31.817082 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.817084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.817250 (+ 166us) server_negotiation.cc:407] Connection header received
0927 21:07:31.817846 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.817847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.817860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.817923 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.818540 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.819329 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820421 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.820682 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820748 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.824317 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.824318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.824319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.824327 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.826588 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826996 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.826996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.826997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.827044 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.827288 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.827290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.827290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.827424 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.827467 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.828584 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.828677 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:31.828892 7589 heartbeater.cc:511] Master 127.0.98.61:35815 requested a full tablet report, sending...
I20260927 21:07:31.829314 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.816963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:47483)
0927 21:07:31.817183 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.817193 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.817732 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.817994 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.817997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.818266 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.818449 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.818453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.819407 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.819408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.820299 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.820301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.820803 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.820804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.820884 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.821526 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.821531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.824205 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.826726 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.826727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.826728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.826916 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.827153 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.827153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.827154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.827214 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.828047 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.828048 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.828452 (+ 404us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.828911 (+ 459us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.828997 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":25,"thread_start_us":80,"threads_started":1}
I20260927 21:07:31.829366 7084 ts_manager.cc:194] Registered new tserver with Master: 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637)
I20260927 21:07:31.829701 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37147
I20260927 21:07:31.829777 7588 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.829838 7588 heartbeater.cc:511] Master 127.0.98.62:37147 requested a full tablet report, sending...
I20260927 21:07:31.830019 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41059
I20260927 21:07:31.830039 6486 ts_manager.cc:194] Re-registered known tserver with Master: 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637)
I20260927 21:07:31.830080 7587 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.830147 7587 heartbeater.cc:511] Master 127.0.98.60:41059 requested a full tablet report, sending...
I20260927 21:07:31.830241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41875
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37147
--webserver_port=45413
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:07:31.830519 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47483
I20260927 21:07:31.830754 7270 ts_manager.cc:194] Registered new tserver with Master: 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637)
W20260927 21:07:31.909185 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:31.909373 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:31.909394 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:31.910501 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:31.910545 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:31.910610 7594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:31.912068 7594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45413
--tserver_master_addrs=127.0.98.62:37147,127.0.98.61:35815,127.0.98.60:41059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:31.912320 7594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:31.912544 7594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:31.917294 7599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:31.917301 7600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.917470 7594 server_base.cc:1034] running on GCE node
W20260927 21:07:31.917368 7602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:31.917786 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:31.918015 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:31.919148 7594 hybrid_clock.cc:648] HybridClock initialized: now 1790543251919144 us; error 36 us; skew 500 ppm
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:31.921590 7594 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:07:31.922039 7594 webserver.cc:460] Webserver started at http://127.0.98.3:45413/ using document root <none> and password file <none>
I20260927 21:07:31.922264 7594 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:31.922345 7594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:31.923712 7594 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:31.924257 7609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.924461 7594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:31.924563 7594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20"
format_stamp: "Formatted at 2026-09-27 21:07:30 on dist-test-slave-8bgs"
server_key: "479382157b096c5a121b8f8fe93b8321"
server_key_iv: "9467494f032782a84f719a7e2553436f"
server_key_version: "encryptionkey@0"
I20260927 21:07:31.924899 7594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:31.939543 7594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:31.941797 7594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:31.941954 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:31.942180 7594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:31.942469 7594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:31.942520 7594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.942561 7594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:31.942588 7594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:31.948493 7594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41875
I20260927 21:07:31.948539 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41875 every 8 connection(s)
I20260927 21:07:31.948873 7594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:31.955024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 7594
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 21:07:31 dist-test-slave-8bgs krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 21:07:31.960332 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.949576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:38575)
0927 21:07:31.949865 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.949873 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.951012 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.951306 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.951310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.951707 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.951899 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.951904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.952604 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.952605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.953885 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.953887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.954239 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.954240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.954368 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.954949 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.954960 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.956710 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.958560 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.958568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.958570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.958875 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.959132 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.959133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.959133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.959267 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.959581 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.959588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.959770 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.959890 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.960081 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":27,"thread_start_us":148,"threads_started":1}
I20260927 21:07:31.960913 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.949708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38575 (local address 127.0.98.62:37147)
0927 21:07:31.949757 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.949759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.949952 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:07:31.951169 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.951170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.951184 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.951217 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.952011 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.952498 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.953969 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.954158 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.954210 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.956812 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.956814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.956814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.956821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.958469 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.958978 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.958980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.958980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.959038 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.959353 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.959354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.959354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.959491 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.959551 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.960730 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.960811 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:07:31.961277 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37147
I20260927 21:07:31.961369 7724 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.961478 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.949965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.98.3:47471)
0927 21:07:31.950088 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.950097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.951025 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.951311 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.951311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.951717 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.951900 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.951906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.953986 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.953987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.954701 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.954703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.956416 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.956418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.956491 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.957030 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.957035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.958545 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.960366 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.960367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.960367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.960615 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.960849 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.960850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.960850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.960903 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.961177 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.961178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.961222 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.961292 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.961389 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260927 21:07:31.961540 7724 heartbeater.cc:511] Master 127.0.98.62:37147 requested a full tablet report, sending...
I20260927 21:07:31.961818 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.950155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47471 (local address 127.0.98.61:35815)
0927 21:07:31.950203 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.950205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.950214 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:31.951153 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.951155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.951167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.951228 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.952017 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.952765 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.954824 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.955150 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.955212 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.958648 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.958649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.958649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.958657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.960278 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.960714 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.960716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.960716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.960774 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.960965 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.960966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.960966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.961119 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.961186 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.961645 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.961723 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:07:31.961871 6486 ts_manager.cc:194] Re-registered known tserver with Master: c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875)
I20260927 21:07:31.961961 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.949576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.98.3:49391)
0927 21:07:31.949799 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.949842 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.951012 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.951312 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.951312 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.951744 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:31.951900 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.951906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.952841 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.952842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.953885 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.953887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.954505 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.954506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.954580 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.955295 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:31.955299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:31.957959 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:31.960256 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.960257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.960257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.960555 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.961050 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:31.961051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:31.961051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:31.961118 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:31.961405 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:31.961405 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:31.961445 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.961505 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.961636 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":6,"thread_start_us":152,"threads_started":1}
I20260927 21:07:31.962359 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35815
I20260927 21:07:31.962402 7725 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.962405 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38575
I20260927 21:07:31.962462 7725 heartbeater.cc:511] Master 127.0.98.61:35815 requested a full tablet report, sending...
I20260927 21:07:31.962502 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.949568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49391 (local address 127.0.98.60:41059)
0927 21:07:31.949646 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.949648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.949974 (+ 326us) server_negotiation.cc:407] Connection header received
0927 21:07:31.951231 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.951233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.951244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.951297 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.952016 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.952756 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.954254 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.954426 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.954504 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.958042 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.958043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.958043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.958050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.960114 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.960631 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.960632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.960633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.960691 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.961191 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.961192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.961192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.961326 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.961381 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.962296 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.962386 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:07:31.962754 7084 ts_manager.cc:194] Registered new tserver with Master: c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875)
I20260927 21:07:31.963146 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41059
I20260927 21:07:31.963236 7723 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:31.963335 7723 heartbeater.cc:511] Master 127.0.98.60:41059 requested a full tablet report, sending...
I20260927 21:07:31.963712 7270 ts_manager.cc:194] Registered new tserver with Master: c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875)
I20260927 21:07:31.966116 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.956526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.0.98.62:37147)
0927 21:07:31.956833 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.956835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.956998 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:07:31.957021 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.957022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.957033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.957104 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.957996 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.958833 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.959991 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.960312 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.960369 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.963432 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.963434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.963434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.963441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.964986 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.965355 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.965356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.965356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.965413 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.965612 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.965613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.965613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.965719 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.965826 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.965929 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.966007 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":79,"threads_started":1}
I20260927 21:07:31.968164 6486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:31.969715 6486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:31.971225 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.959523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.0.98.60:41059)
0927 21:07:31.962573 (+ 3050us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.962575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.963408 (+ 833us) server_negotiation.cc:407] Connection header received
0927 21:07:31.963577 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.963578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.963589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.963619 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.964200 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.964906 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.966527 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.966820 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.966876 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.968448 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.968449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.968450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.968456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.969973 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.970371 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.970372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.970372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.970427 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.970647 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.970648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.970648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.970797 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.970847 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.971025 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.971111 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3035,"thread_start_us":66,"threads_started":1}
I20260927 21:07:31.971988 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.956913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.0.98.61:35815)
0927 21:07:31.961901 (+ 4988us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.961903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.963415 (+ 1512us) server_negotiation.cc:407] Connection header received
0927 21:07:31.963519 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.963520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.963531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.963573 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:31.964600 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.965330 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.966388 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.966666 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.966733 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.968288 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:31.968289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:31.968289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:31.968296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:31.970654 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.971041 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.971042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.971042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.971098 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:31.971354 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:31.971356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:31.971356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:31.971497 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:31.971582 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.971751 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.971817 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4973,"thread_start_us":47,"threads_started":1}
I20260927 21:07:31.977581 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.973602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40637 (local address 127.0.0.1:59476)
0927 21:07:31.973667 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.973678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.973726 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.974184 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.974184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.974186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:31.974221 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.974222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.975475 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.975476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.976499 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.976501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.977385 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.977385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.977427 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.977431 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.977463 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.977490 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":63,"thread_start_us":39,"threads_started":1}
I20260927 21:07:31.977614 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.973896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.98.2:40637)
0927 21:07:31.974050 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.974054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.974063 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:31.974098 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.974100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.974116 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.974189 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:31.974273 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.975385 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.976581 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.977295 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.977382 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.977449 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.977476 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.977522 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260927 21:07:31.977813 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.973759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41875 (local address 127.0.0.1:47118)
0927 21:07:31.973863 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.973873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.973918 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.974378 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.974378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.974380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:31.974407 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.974408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.975588 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.975588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.976712 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.976714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.977553 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.977554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.977610 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.977614 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.977712 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.977755 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":57,"thread_start_us":70,"threads_started":1}
I20260927 21:07:31.978268 7521 tablet_service.cc:1467] Processing CreateTablet for tablet afddd880236d4bc096c97731c1a531bc (DEFAULT_TABLE table=test-table [id=da1e462f76d54aeab9c7c1ded67ff7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:31.978636 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afddd880236d4bc096c97731c1a531bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:31.979056 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.973773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.0.98.1:35131)
0927 21:07:31.974282 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.974286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.974295 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:31.974331 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.974333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.974348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.974444 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:31.974553 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.975375 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.977847 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.978491 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.978611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.978673 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.978926 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.978978 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":96,"threads_started":1}
I20260927 21:07:31.981413 7751 tablet_bootstrap.cc:492] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Bootstrap starting.
I20260927 21:07:31.978863 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.973692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:33970)
0927 21:07:31.973816 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:31.973825 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:31.973862 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:31.974445 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:31.974446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:31.974448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:31.974498 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.974500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.975458 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.975459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.976712 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:31.976714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.978619 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.978620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:31.978692 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.978697 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:07:31.978754 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:07:31.978804 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":56,"thread_start_us":90,"threads_started":1}
I20260927 21:07:31.981817 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:31.974102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.0.98.3:41875)
0927 21:07:31.974236 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:31.974241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:31.974249 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:31.974283 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:31.974285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:31.974300 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:31.974383 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:31.974483 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.975516 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.976798 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:31.977467 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:31.977540 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:31.977600 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:31.981674 (+ 4074us) server_negotiation.cc:299] Negotiation successful
0927 21:07:31.981724 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260927 21:07:31.982095 7751 tablet_bootstrap.cc:654] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:31.982120 7385 tablet_service.cc:1467] Processing CreateTablet for tablet afddd880236d4bc096c97731c1a531bc (DEFAULT_TABLE table=test-table [id=da1e462f76d54aeab9c7c1ded67ff7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:31.982399 7751 log.cc:824] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:31.982479 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afddd880236d4bc096c97731c1a531bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:31.982399 7657 tablet_service.cc:1467] Processing CreateTablet for tablet afddd880236d4bc096c97731c1a531bc (DEFAULT_TABLE table=test-table [id=da1e462f76d54aeab9c7c1ded67ff7e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:31.982688 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afddd880236d4bc096c97731c1a531bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:31.983213 7751 tablet_bootstrap.cc:492] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: No bootstrap required, opened a new log
I20260927 21:07:31.983281 7751 ts_tablet_manager.cc:1397] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:31.985234 7753 tablet_bootstrap.cc:492] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Bootstrap starting.
I20260927 21:07:31.985745 7753 tablet_bootstrap.cc:654] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:31.986027 7753 log.cc:824] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:31.986202 7751 raft_consensus.cc:348] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:31.986371 7751 raft_consensus.cc:374] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:31.986392 7755 tablet_bootstrap.cc:492] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Bootstrap starting.
I20260927 21:07:31.986421 7751 raft_consensus.cc:729] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f3f863734b4118ab156dda40c0811a, State: Initialized, Role: FOLLOWER
I20260927 21:07:31.986559 7751 consensus_queue.cc:260] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:31.986837 7751 ts_tablet_manager.cc:1428] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:07:31.986912 7753 tablet_bootstrap.cc:492] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: No bootstrap required, opened a new log
I20260927 21:07:31.986976 7753 ts_tablet_manager.cc:1397] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:31.986979 7755 tablet_bootstrap.cc:654] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:31.987193 7588 heartbeater.cc:503] Master 127.0.98.62:37147 was elected leader, sending a full tablet report...
I20260927 21:07:31.987313 7755 log.cc:824] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:31.987987 7755 tablet_bootstrap.cc:492] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: No bootstrap required, opened a new log
I20260927 21:07:31.988049 7755 ts_tablet_manager.cc:1397] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:07:31.989959 7753 raft_consensus.cc:348] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:31.990077 7753 raft_consensus.cc:374] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:31.990098 7753 raft_consensus.cc:729] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf43902fdf141bc8c06e65a5cf881d7, State: Initialized, Role: FOLLOWER
I20260927 21:07:31.990240 7753 consensus_queue.cc:260] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:31.990270 7755 raft_consensus.cc:348] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:31.990460 7755 raft_consensus.cc:374] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:31.990501 7755 raft_consensus.cc:729] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdc8ad18fa48d7997b4d3da4b31a20, State: Initialized, Role: FOLLOWER
I20260927 21:07:31.990612 7755 consensus_queue.cc:260] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:31.990881 7755 ts_tablet_manager.cc:1428] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:07:31.990906 7753 ts_tablet_manager.cc:1428] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:07:31.990957 7724 heartbeater.cc:503] Master 127.0.98.62:37147 was elected leader, sending a full tablet report...
I20260927 21:07:31.990986 7452 heartbeater.cc:503] Master 127.0.98.62:37147 was elected leader, sending a full tablet report...
I20260927 21:07:32.046991 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.044335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.0.98.60:41059)
0927 21:07:32.044389 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:32.044391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:32.044415 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:07:32.044513 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:32.044514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:32.044535 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:32.044587 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:32.044691 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.045295 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.046367 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.046750 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.046809 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.046867 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:32.046910 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:07:32.046934 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:07:32.047000 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.044267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.0.1:51030)
0927 21:07:32.044366 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:32.044378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:32.044433 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:32.044591 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:32.044592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:32.044593 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:32.044638 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:32.044640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.045392 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.045392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:32.046293 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:32.046295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.046820 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.046821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:32.046863 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.046866 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:32.046905 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 21:07:32.046942 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:07:32.047375 7280 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 21:07:32.048293 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243a548bacc447189e56dc242752e0fc. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:32.048403 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: This master's current role is: LEARNER
W20260927 21:07:32.067022 7590 tablet.cc:2369] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:32.102226 7759 raft_consensus.cc:482] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:32.102341 7759 raft_consensus.cc:504] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:32.102731 7759 leader_election.cc:288] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637), ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131)
I20260927 21:07:32.105849 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.103063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53665 (local address 127.0.98.1:35131)
0927 21:07:32.103140 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:32.103142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:32.103152 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:32.103217 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:32.103218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:32.103230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:32.103288 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:32.103459 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.104035 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.105134 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.105523 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.105606 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.105615 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:32.105738 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:07:32.105779 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:07:32.105871 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.103010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40613 (local address 127.0.98.2:40637)
0927 21:07:32.103057 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:32.103059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:32.103072 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:07:32.103148 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:32.103149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:32.103161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:32.103221 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:32.103382 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.103967 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.105156 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.105524 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.105610 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.105619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:32.105778 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 21:07:32.105813 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:07:32.105870 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.102898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.98.3:53665)
0927 21:07:32.103058 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:32.103069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:32.103138 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:32.103298 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:32.103299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:32.103300 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:32.103373 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:32.103374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.104111 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.104112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:32.105021 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:32.105024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.105605 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.105606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:32.105688 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.105691 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:32.105730 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 21:07:32.105785 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":72}
I20260927 21:07:32.105890 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.102898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40637 (local address 127.0.98.3:40613)
0927 21:07:32.102960 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:32.102972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:32.103058 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:32.103234 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:32.103235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:32.103237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:32.103309 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:32.103311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.104035 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.104036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:32.105030 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:32.105032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.105613 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.105614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:32.105704 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.105707 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:32.105761 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:07:32.105797 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":29}
I20260927 21:07:32.106195 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afddd880236d4bc096c97731c1a531bc" candidate_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" is_pre_election: true
I20260927 21:07:32.106225 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afddd880236d4bc096c97731c1a531bc" candidate_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f3f863734b4118ab156dda40c0811a" is_pre_election: true
I20260927 21:07:32.106353 7541 raft_consensus.cc:2454] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4fdc8ad18fa48d7997b4d3da4b31a20 in term 0.
I20260927 21:07:32.106348 7405 raft_consensus.cc:2454] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4fdc8ad18fa48d7997b4d3da4b31a20 in term 0.
I20260927 21:07:32.106560 7613 leader_election.cc:302] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f3f863734b4118ab156dda40c0811a, c4fdc8ad18fa48d7997b4d3da4b31a20; no voters:
I20260927 21:07:32.106690 7759 raft_consensus.cc:2781] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:32.106742 7759 raft_consensus.cc:482] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:32.106765 7759 raft_consensus.cc:3037] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:32.107471 7759 raft_consensus.cc:504] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:32.107604 7759 leader_election.cc:288] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [CANDIDATE]: Term 1 election: Requested vote from peers 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637), ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131)
I20260927 21:07:32.107726 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afddd880236d4bc096c97731c1a531bc" candidate_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf43902fdf141bc8c06e65a5cf881d7"
I20260927 21:07:32.107741 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afddd880236d4bc096c97731c1a531bc" candidate_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f3f863734b4118ab156dda40c0811a"
I20260927 21:07:32.107848 7541 raft_consensus.cc:3037] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:32.107851 7405 raft_consensus.cc:3037] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:32.108563 7541 raft_consensus.cc:2454] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4fdc8ad18fa48d7997b4d3da4b31a20 in term 1.
I20260927 21:07:32.108577 7405 raft_consensus.cc:2454] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4fdc8ad18fa48d7997b4d3da4b31a20 in term 1.
I20260927 21:07:32.108727 7611 leader_election.cc:302] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4fdc8ad18fa48d7997b4d3da4b31a20, ecf43902fdf141bc8c06e65a5cf881d7; no voters:
I20260927 21:07:32.108850 7759 raft_consensus.cc:2781] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:32.109014 7759 raft_consensus.cc:686] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 LEADER]: Becoming Leader. State: Replica: c4fdc8ad18fa48d7997b4d3da4b31a20, State: Running, Role: LEADER
I20260927 21:07:32.109133 7759 consensus_queue.cc:237] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:32.109886 6486 catalog_manager.cc:5495] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } health_report { overall_health: HEALTHY } } }
I20260927 21:07:32.110536 7742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: true } } }
I20260927 21:07:32.110646 7742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:32.111305 7234 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: attempting to promote NON_VOTER 60f44e091d9d4c5a87fa0d21c2343e25 to VOTER
I20260927 21:07:32.111429 7234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } }
I20260927 21:07:32.111982 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Refusing update from remote peer 243a548bacc447189e56dc242752e0fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 21:07:32.112087 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 LEARNER]: Refusing update from remote peer 243a548bacc447189e56dc242752e0fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 21:07:32.112172 7767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 21:07:32.112267 7770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 21:07:32.112847 7770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.113078 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.113128 7280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.113730 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.113804 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:32.113858 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.113927 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:32.114027 7768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.114106 7768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:32.114338 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.114408 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:07:32.114517 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "243a548bacc447189e56dc242752e0fc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:32.114576 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:07:32.119948 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.118016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.0.98.3:41875)
0927 21:07:32.118049 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:32.118051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:32.118059 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:32.118109 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:32.118109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:32.118123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:32.118180 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:32.118263 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.118670 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.119368 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.119554 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.119604 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.119722 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:32.119804 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:32.119860 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:32.119890 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 21:07:32.119905 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 21:07:32.121440 7405 raft_consensus.cc:1261] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Refusing update from remote peer c4fdc8ad18fa48d7997b4d3da4b31a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:32.121452 7541 raft_consensus.cc:1261] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Refusing update from remote peer c4fdc8ad18fa48d7997b4d3da4b31a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:32.121770 7759 consensus_queue.cc:1035] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:32.121899 7765 consensus_queue.cc:1035] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:32.124027 7773 mvcc.cc:205] Tried to move back new op lower bound from 7334065160687370240 to 7334065160639672320. Current Snapshot: MvccSnapshot[applied={T|T < 7334065160687370240}]
I20260927 21:07:32.124910 7775 mvcc.cc:205] Tried to move back new op lower bound from 7334065160687370240 to 7334065160639672320. Current Snapshot: MvccSnapshot[applied={T|T < 7334065160687370240}]
I20260927 21:07:32.126263 6485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:32.126344 6485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:32.126462 7774 mvcc.cc:205] Tried to move back new op lower bound from 7334065160687370240 to 7334065160639672320. Current Snapshot: MvccSnapshot[applied={T|T < 7334065160687370240}]
I20260927 21:07:32.127910 6485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 243a548bacc447189e56dc242752e0fc: Sending DeleteTablet for 3 replicas of tablet afddd880236d4bc096c97731c1a531bc
I20260927 21:07:32.128250 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet afddd880236d4bc096c97731c1a531bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47118
I20260927 21:07:32.128334 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet afddd880236d4bc096c97731c1a531bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59476
I20260927 21:07:32.128943 7782 tablet_replica.cc:330] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: stopping tablet replica
I20260927 21:07:32.129051 7782 raft_consensus.cc:2229] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:32.129142 7782 raft_consensus.cc:2258] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:32.129276 7783 tablet_replica.cc:330] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: stopping tablet replica
I20260927 21:07:32.129366 7783 raft_consensus.cc:2229] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:32.129463 7783 raft_consensus.cc:2258] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:32.129732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6454
I20260927 21:07:32.129735 7782 ts_tablet_manager.cc:1905] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:32.129820 7783 ts_tablet_manager.cc:1905] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:32.131469 7783 ts_tablet_manager.cc:1918] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:32.131469 7782 ts_tablet_manager.cc:1918] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:32.131541 7783 log.cc:1196] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/afddd880236d4bc096c97731c1a531bc
I20260927 21:07:32.131634 7782 log.cc:1196] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/afddd880236d4bc096c97731c1a531bc
I20260927 21:07:32.131819 7783 ts_tablet_manager.cc:1939] T afddd880236d4bc096c97731c1a531bc P 10f3f863734b4118ab156dda40c0811a: Deleting consensus metadata
I20260927 21:07:32.132005 7782 ts_tablet_manager.cc:1939] T afddd880236d4bc096c97731c1a531bc P c4fdc8ad18fa48d7997b4d3da4b31a20: Deleting consensus metadata
I20260927 21:07:32.132248 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet afddd880236d4bc096c97731c1a531bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33970
I20260927 21:07:32.135246 7784 tablet_replica.cc:330] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: stopping tablet replica
I20260927 21:07:32.135353 7784 raft_consensus.cc:2229] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:32.135447 7784 raft_consensus.cc:2258] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:32.135823 7784 ts_tablet_manager.cc:1905] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:32.137100 7784 ts_tablet_manager.cc:1918] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:32.137180 7784 log.cc:1196] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/afddd880236d4bc096c97731c1a531bc
W20260927 21:07:32.137363 7613 connection.cc:535] server connection from 127.0.0.1:47118 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:07:32.137487 7784 ts_tablet_manager.cc:1939] T afddd880236d4bc096c97731c1a531bc P ecf43902fdf141bc8c06e65a5cf881d7: Deleting consensus metadata
W20260927 21:07:32.137573 7476 connection.cc:535] server connection from 127.0.0.1:59476 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:07:32.137847 7339 connection.cc:441] server connection from 127.0.0.1:33970 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33970 (request call id 1) could send its response
I20260927 21:07:32.145982 7129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.139230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.0.98.61:35815)
0927 21:07:32.139273 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:32.139275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:32.139407 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:07:32.139518 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:32.139519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:32.139530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:32.139584 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:32.140088 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.140560 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.141271 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.141478 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.141560 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.142927 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:32.142929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:32.142930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:32.142937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:32.144676 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:32.145074 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:32.145076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:32.145076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:32.145133 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:32.145365 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:32.145366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:32.145367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:32.145514 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:32.145586 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:32.145811 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:07:32.145885 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:32.146919 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:32.139450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.0.98.60:41059)
0927 21:07:32.139489 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:32.139491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:32.139499 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:32.139520 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:32.139521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:32.139530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:32.139583 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:32.140307 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.141018 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.142055 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:32.142345 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:32.142382 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:32.144040 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:32.144042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:32.144042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:32.144050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:32.145875 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:32.146245 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:32.146247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:32.146247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:32.146300 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:32.146485 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:32.146486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:32.146486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:32.146599 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:32.146669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:32.146801 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 21:07:32.146865 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:33.126616 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.126316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:43775)
0927 21:07:33.126447 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.126548 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":100,"threads_started":1}
W20260927 21:07:33.126933 7724 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.129268 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.129056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:58241)
0927 21:07:33.129166 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.129235 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260927 21:07:33.129464 7452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.132507 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.132329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:54633)
0927 21:07:33.132422 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.132474 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
W20260927 21:07:33.132705 7588 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.628386 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 243a548bacc447189e56dc242752e0fc)
I20260927 21:07:33.628376 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 243a548bacc447189e56dc242752e0fc)
I20260927 21:07:33.628535 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } }
I20260927 21:07:33.628530 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } }
I20260927 21:07:33.628837 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147), 3e3ff17a6a4249d3b3640b8dc0435a19 (127.0.98.61:35815)
I20260927 21:07:33.629156 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.628907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42478)
0927 21:07:33.629045 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.629110 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":45,"threads_started":1}
I20260927 21:07:33.629194 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147), 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60:41059)
W20260927 21:07:33.629300 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.629395 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.628909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42488)
0927 21:07:33.629299 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.629356 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":66,"threads_started":1}
W20260927 21:07:33.629501 7254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.629911 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.629812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42498)
0927 21:07:33.629842 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.629873 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 21:07:33.629992 7254 leader_election.cc:334] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.630008 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.629885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42512)
0927 21:07:33.629922 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.629944 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 21:07:33.630100 7068 leader_election.cc:334] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.632001 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.628932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41059 (local address 127.0.0.1:51050)
0927 21:07:33.629070 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.629081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:33.629147 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:33.629372 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:33.629373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:33.629375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:33.629431 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.629433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.630387 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.630388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.631212 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.631214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.631826 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.631827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.631875 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.631879 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:33.631920 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 21:07:33.631956 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":43,"threads_started":1}
I20260927 21:07:33.632030 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.629040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.0.98.60:41059)
0927 21:07:33.629212 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:33.629215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:33.629224 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:33.629248 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:33.629249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:33.629260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:33.629347 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:33.629517 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.630298 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.631302 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.631750 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.631820 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.631831 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:33.631938 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:07:33.631972 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":38,"threads_started":1}
I20260927 21:07:33.632215 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" is_pre_election: true
I20260927 21:07:33.632314 7280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3ff17a6a4249d3b3640b8dc0435a19 in term 1.
I20260927 21:07:33.632463 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.629092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35815 (local address 127.0.0.1:54406)
0927 21:07:33.629678 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.629688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:33.629871 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:33.630071 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:33.630072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:33.630074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:33.630113 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.630115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.630745 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.630746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.631545 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.631547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.632239 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.632240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.632303 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.632308 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:07:33.632358 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:07:33.632395 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":44,"threads_started":1}
I20260927 21:07:33.632452 7069 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e3ff17a6a4249d3b3640b8dc0435a19, 60f44e091d9d4c5a87fa0d21c2343e25; no voters: 243a548bacc447189e56dc242752e0fc
I20260927 21:07:33.632534 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.629132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.0.98.61:35815)
0927 21:07:33.629409 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:33.629411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:33.629846 (+ 435us) server_negotiation.cc:407] Connection header received
0927 21:07:33.629949 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:33.629950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:33.629960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:33.630007 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:33.630188 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.630671 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.631622 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.632170 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.632240 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.632251 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:33.632387 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:07:33.632433 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":33,"threads_started":1}
I20260927 21:07:33.632586 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:07:33.632615 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 243a548bacc447189e56dc242752e0fc)
I20260927 21:07:33.632634 7795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:07:33.632741 7093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" is_pre_election: true
I20260927 21:07:33.633400 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } }
I20260927 21:07:33.633524 7093 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60f44e091d9d4c5a87fa0d21c2343e25 in current term 2: Already voted for candidate 3e3ff17a6a4249d3b3640b8dc0435a19 in this term.
I20260927 21:07:33.633545 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [CANDIDATE]: Term 2 election: Requested vote from peers 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147), 60f44e091d9d4c5a87fa0d21c2343e25 (127.0.98.60:41059)
I20260927 21:07:33.633719 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.633608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42524)
0927 21:07:33.633670 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.633694 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:07:33.633718 7255 leader_election.cc:302] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60f44e091d9d4c5a87fa0d21c2343e25; no voters: 243a548bacc447189e56dc242752e0fc, 3e3ff17a6a4249d3b3640b8dc0435a19
I20260927 21:07:33.633734 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60f44e091d9d4c5a87fa0d21c2343e25"
I20260927 21:07:33.633800 7280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:07:33.633987 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.633914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42528)
0927 21:07:33.633948 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.633968 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 21:07:33.634093 7068 leader_election.cc:334] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
I20260927 21:07:33.634498 7280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3ff17a6a4249d3b3640b8dc0435a19 in term 2.
I20260927 21:07:33.634569 7794 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:07:33.634656 7069 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e3ff17a6a4249d3b3640b8dc0435a19, 60f44e091d9d4c5a87fa0d21c2343e25; no voters: 243a548bacc447189e56dc242752e0fc
I20260927 21:07:33.634750 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:07:33.634866 7795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [term 2 LEADER]: Becoming Leader. State: Replica: 3e3ff17a6a4249d3b3640b8dc0435a19, State: Running, Role: LEADER
I20260927 21:07:33.634954 7795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } }
I20260927 21:07:33.635285 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3ff17a6a4249d3b3640b8dc0435a19. Latest consensus state: current_term: 2 leader_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:33.635381 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:33.635545 7806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:33.635939 7806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:33.635996 7806 catalog_manager.cc:1259] Loaded cluster ID: 2b9575b6379c40a998af10a534adb1f6
I20260927 21:07:33.636044 7806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:33.636191 7806 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:33.636272 7806 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Loaded TSK: 0
I20260927 21:07:33.636401 7806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:33.677683 7083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:33.679306 7083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:33.680402 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.680258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42548)
0927 21:07:33.680322 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.680372 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 21:07:33.680433 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [term 2 FOLLOWER]: Refusing update from remote peer 3e3ff17a6a4249d3b3640b8dc0435a19: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 21:07:33.680732 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.680627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.0.1:42564)
0927 21:07:33.680659 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.680685 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 21:07:33.680742 7804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 21:07:33.680871 7068 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 -> Peer 243a548bacc447189e56dc242752e0fc (127.0.98.62:37147): Couldn't send request to peer 243a548bacc447189e56dc242752e0fc. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:07:33.681384 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3ff17a6a4249d3b3640b8dc0435a19. Latest consensus state: current_term: 2 leader_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:33.681484 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:07:33.681914 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:33.682000 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:33.682029 7808 mvcc.cc:205] Tried to move back new op lower bound from 7334065167072600064 to 7334065166889299968. Current Snapshot: MvccSnapshot[applied={T|T < 7334065167072600064}]
I20260927 21:07:33.682142 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "243a548bacc447189e56dc242752e0fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37147 } } peers { permanent_uuid: "3e3ff17a6a4249d3b3640b8dc0435a19" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35815 } attrs { promote: false } } peers { permanent_uuid: "60f44e091d9d4c5a87fa0d21c2343e25" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41059 } attrs { promote: false } } }
I20260927 21:07:33.682225 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f44e091d9d4c5a87fa0d21c2343e25 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:07:33.687021 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.683753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41875 (local address 127.0.0.1:47136)
0927 21:07:33.683879 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.683887 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:33.683925 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:33.684145 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:33.684146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:33.684148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:33.684197 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.684198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.685110 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.685111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.686148 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.686151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.686787 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.686846 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.686850 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:33.686896 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:07:33.686933 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260927 21:07:33.687148 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.683697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40637 (local address 127.0.0.1:59482)
0927 21:07:33.683744 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.683753 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:33.683793 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:33.684029 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:33.684029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:33.684032 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:33.684123 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.684125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.685033 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.685034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.686119 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.686120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.686875 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.686942 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.686945 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:33.686976 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 21:07:33.687021 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:07:33.687239 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.683870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.98.1:35131)
0927 21:07:33.683960 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:33.683962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:33.683971 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:33.684013 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:33.684013 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:33.684023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:33.684104 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:33.684208 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.684798 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686214 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.686775 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686877 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.686889 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:33.687008 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 21:07:33.687053 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":49,"threads_started":1}
I20260927 21:07:33.687244 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.683753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.0.98.2:40637)
0927 21:07:33.683892 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:33.683894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:33.683903 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:33.683930 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:33.683931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:33.683941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:33.684188 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:33.684227 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.684958 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686217 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.686775 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.687065 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.687079 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:33.687120 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 21:07:33.687168 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":51,"threads_started":1}
I20260927 21:07:33.687296 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.683811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.98.3:41875)
0927 21:07:33.684018 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:33.684020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:33.684028 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:33.684051 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:33.684051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:33.684062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:33.684294 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:33.684332 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.685050 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686243 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.686678 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.686799 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:33.687141 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 21:07:33.687210 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":51,"threads_started":1}
I20260927 21:07:33.687513 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9b00fdf91045689670edf6900d84a7 (DEFAULT_TABLE table=test-table [id=13715f8c376c48509d8e5bed8caa26a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:33.687661 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9b00fdf91045689670edf6900d84a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:33.687901 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.683625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:33976)
0927 21:07:33.683652 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.683661 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:33.683699 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:33.684107 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:33.684108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:33.684110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:33.684154 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.684155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.684879 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.684880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.686106 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:33.686108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:33.686878 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:33.686878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:33.686933 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:33.686935 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:33.686985 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:07:33.687022 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 21:07:33.687904 7521 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9b00fdf91045689670edf6900d84a7 (DEFAULT_TABLE table=test-table [id=13715f8c376c48509d8e5bed8caa26a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:33.688037 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9b00fdf91045689670edf6900d84a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:33.688163 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9b00fdf91045689670edf6900d84a7 (DEFAULT_TABLE table=test-table [id=13715f8c376c48509d8e5bed8caa26a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:33.688279 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9b00fdf91045689670edf6900d84a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:33.689124 7817 tablet_bootstrap.cc:492] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Bootstrap starting.
I20260927 21:07:33.689498 7818 tablet_bootstrap.cc:492] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Bootstrap starting.
I20260927 21:07:33.689628 7817 tablet_bootstrap.cc:654] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:33.689874 7818 tablet_bootstrap.cc:654] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:33.689952 7819 tablet_bootstrap.cc:492] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: Bootstrap starting.
I20260927 21:07:33.690259 7817 tablet_bootstrap.cc:492] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: No bootstrap required, opened a new log
I20260927 21:07:33.690310 7819 tablet_bootstrap.cc:654] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:33.690315 7817 ts_tablet_manager.cc:1397] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:33.690387 7818 tablet_bootstrap.cc:492] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: No bootstrap required, opened a new log
I20260927 21:07:33.690439 7818 ts_tablet_manager.cc:1397] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:33.690433 7817 raft_consensus.cc:348] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:33.690512 7817 raft_consensus.cc:374] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:33.690542 7817 raft_consensus.cc:729] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fdc8ad18fa48d7997b4d3da4b31a20, State: Initialized, Role: FOLLOWER
I20260927 21:07:33.690568 7818 raft_consensus.cc:348] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:33.690641 7818 raft_consensus.cc:374] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:33.690630 7817 consensus_queue.cc:260] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:33.690668 7818 raft_consensus.cc:729] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f3f863734b4118ab156dda40c0811a, State: Initialized, Role: FOLLOWER
I20260927 21:07:33.690727 7818 consensus_queue.cc:260] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:33.690773 7817 ts_tablet_manager.cc:1428] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:33.690790 7819 tablet_bootstrap.cc:492] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: No bootstrap required, opened a new log
I20260927 21:07:33.690828 7819 ts_tablet_manager.cc:1397] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:33.690972 7819 raft_consensus.cc:348] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:33.691043 7819 raft_consensus.cc:374] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:33.691062 7819 raft_consensus.cc:729] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf43902fdf141bc8c06e65a5cf881d7, State: Initialized, Role: FOLLOWER
I20260927 21:07:33.691134 7819 consensus_queue.cc:260] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:33.691289 7819 ts_tablet_manager.cc:1428] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:33.692032 7725 heartbeater.cc:503] Master 127.0.98.61:35815 was elected leader, sending a full tablet report...
I20260927 21:07:33.692258 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.691061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:38599)
0927 21:07:33.692156 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.692196 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":304,"threads_started":1}
I20260927 21:07:33.692435 7453 heartbeater.cc:503] Master 127.0.98.61:35815 was elected leader, sending a full tablet report...
I20260927 21:07:33.692559 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.691505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:51385)
0927 21:07:33.692459 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.692519 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":71,"threads_started":1}
I20260927 21:07:33.692220 7818 ts_tablet_manager.cc:1428] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:33.692647 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.691482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:54545)
0927 21:07:33.692361 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.692395 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":69,"threads_started":1}
I20260927 21:07:33.692829 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.692535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:41587)
0927 21:07:33.692775 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.692805 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260927 21:07:33.693102 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.692906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:45751)
0927 21:07:33.693021 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.693057 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260927 21:07:33.693171 7589 heartbeater.cc:503] Master 127.0.98.61:35815 was elected leader, sending a full tablet report...
I20260927 21:07:33.693248 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:33.693081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:46819)
0927 21:07:33.693114 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:33.693204 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 21:07:33.700821 7726 tablet.cc:2369] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:07:33.937917 7454 tablet.cc:2369] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:34.029314 7825 raft_consensus.cc:482] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:34.029412 7825 raft_consensus.cc:504] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:34.029764 7825 leader_election.cc:288] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637), c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875)
I20260927 21:07:34.032666 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.030054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45183 (local address 127.0.98.3:41875)
0927 21:07:34.030114 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:34.030115 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:34.030125 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:34.030200 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:34.030201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:34.030220 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:34.030284 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:34.030447 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.030923 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.031944 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.032301 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.032383 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:34.032395 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:34.032548 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:07:34.032584 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:07:34.032678 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.030052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52591 (local address 127.0.98.2:40637)
0927 21:07:34.030094 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:34.030096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:34.030109 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:07:34.030197 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:34.030198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:34.030209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:34.030277 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:34.030448 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.030927 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.031949 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.032314 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.032395 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:34.032407 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:34.032565 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:07:34.032608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:07:34.032689 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.029962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40637 (local address 127.0.98.1:52591)
0927 21:07:34.030027 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.030041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:34.030126 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:34.030281 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:34.030282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:34.030284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:34.030376 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:34.030377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.031009 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.031010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:34.031856 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:34.031858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.032420 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.032421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:34.032491 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:34.032494 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:34.032560 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:07:34.032597 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":78}
I20260927 21:07:34.032702 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.029962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41875 (local address 127.0.98.1:45183)
0927 21:07:34.030056 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.030068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:34.030133 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:34.030286 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:34.030287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:34.030289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:34.030373 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:34.030375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.031011 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.031012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:34.031843 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:34.031845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.032383 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.032384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:34.032461 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:34.032465 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:34.032540 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:07:34.032577 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":55}
I20260927 21:07:34.032936 7677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9b00fdf91045689670edf6900d84a7" candidate_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" is_pre_election: true
I20260927 21:07:34.033097 7677 raft_consensus.cc:2454] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf43902fdf141bc8c06e65a5cf881d7 in term 0.
I20260927 21:07:34.033205 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9b00fdf91045689670edf6900d84a7" candidate_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f3f863734b4118ab156dda40c0811a" is_pre_election: true
I20260927 21:07:34.033279 7541 raft_consensus.cc:2454] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf43902fdf141bc8c06e65a5cf881d7 in term 0.
I20260927 21:07:34.033303 7338 leader_election.cc:302] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4fdc8ad18fa48d7997b4d3da4b31a20, ecf43902fdf141bc8c06e65a5cf881d7; no voters:
I20260927 21:07:34.033468 7825 raft_consensus.cc:2781] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:34.033514 7825 raft_consensus.cc:482] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:34.033527 7825 raft_consensus.cc:3037] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:34.034186 7825 raft_consensus.cc:504] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:34.034332 7825 leader_election.cc:288] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [CANDIDATE]: Term 1 election: Requested vote from peers 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637), c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875)
I20260927 21:07:34.034500 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9b00fdf91045689670edf6900d84a7" candidate_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f3f863734b4118ab156dda40c0811a"
I20260927 21:07:34.034595 7541 raft_consensus.cc:3037] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:34.035318 7541 raft_consensus.cc:2454] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf43902fdf141bc8c06e65a5cf881d7 in term 1.
I20260927 21:07:34.035446 7677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9b00fdf91045689670edf6900d84a7" candidate_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20"
I20260927 21:07:34.035526 7677 raft_consensus.cc:3037] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:34.035502 7341 leader_election.cc:302] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f3f863734b4118ab156dda40c0811a, ecf43902fdf141bc8c06e65a5cf881d7; no voters:
I20260927 21:07:34.035605 7825 raft_consensus.cc:2781] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:34.035760 7825 raft_consensus.cc:686] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 LEADER]: Becoming Leader. State: Replica: ecf43902fdf141bc8c06e65a5cf881d7, State: Running, Role: LEADER
I20260927 21:07:34.035943 7825 consensus_queue.cc:237] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } }
I20260927 21:07:34.036422 7677 raft_consensus.cc:2454] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf43902fdf141bc8c06e65a5cf881d7 in term 1.
I20260927 21:07:34.036809 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.036669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:35917)
0927 21:07:34.036727 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.036759 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 21:07:34.036916 7083 catalog_manager.cc:5495] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf43902fdf141bc8c06e65a5cf881d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:34.126395 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.124064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.0.98.1:35131)
0927 21:07:34.124133 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:34.124135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:34.124144 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:34.124194 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:34.124194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:34.124215 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:34.124294 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:34.124427 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.124907 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.125638 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:34.125831 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:34.125913 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:34.126087 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:34.126182 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:34.126262 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:34.126293 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 21:07:34.126314 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:07:34.127614 7541 raft_consensus.cc:1261] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Refusing update from remote peer ecf43902fdf141bc8c06e65a5cf881d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:34.127638 7677 raft_consensus.cc:1261] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Refusing update from remote peer ecf43902fdf141bc8c06e65a5cf881d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:34.127948 7830 consensus_queue.cc:1035] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f3f863734b4118ab156dda40c0811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:34.128073 7825 consensus_queue.cc:1035] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fdc8ad18fa48d7997b4d3da4b31a20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:34.128963 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.128628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:37405)
0927 21:07:34.128663 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.128699 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 21:07:34.129501 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.129369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:44457)
0927 21:07:34.129432 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.129468 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:07:34.130409 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.130298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.1:52109)
0927 21:07:34.130339 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.130366 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 21:07:34.130774 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.130623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.3:47367)
0927 21:07:34.130718 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.130745 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 21:07:34.130964 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.129969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37147 (local address 127.0.98.2:36607)
0927 21:07:34.130129 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.130937 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 21:07:34.132946 7083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:34.133038 7083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:34.134279 7083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e3ff17a6a4249d3b3640b8dc0435a19: Sending DeleteTablet for 3 replicas of tablet 4f9b00fdf91045689670edf6900d84a7
I20260927 21:07:34.134575 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9b00fdf91045689670edf6900d84a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59482
I20260927 21:07:34.134570 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9b00fdf91045689670edf6900d84a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47136
I20260927 21:07:34.134727 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9b00fdf91045689670edf6900d84a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33976
I20260927 21:07:34.134752 7844 tablet_replica.cc:330] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: stopping tablet replica
I20260927 21:07:34.134843 7844 raft_consensus.cc:2229] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:34.134904 7844 raft_consensus.cc:2258] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:34.134927 7846 tablet_replica.cc:330] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7: stopping tablet replica
I20260927 21:07:34.134985 7846 raft_consensus.cc:2229] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:34.135195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 7322
I20260927 21:07:34.135213 7846 raft_consensus.cc:2258] T 4f9b00fdf91045689670edf6900d84a7 P ecf43902fdf141bc8c06e65a5cf881d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:34.136139 7844 ts_tablet_manager.cc:1905] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:34.138109 7844 ts_tablet_manager.cc:1918] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:34.138170 7844 log.cc:1196] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/4f9b00fdf91045689670edf6900d84a7
I20260927 21:07:34.138271 7845 tablet_replica.cc:330] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: stopping tablet replica
I20260927 21:07:34.138352 7845 raft_consensus.cc:2229] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:34.138394 7844 ts_tablet_manager.cc:1939] T 4f9b00fdf91045689670edf6900d84a7 P 10f3f863734b4118ab156dda40c0811a: Deleting consensus metadata
I20260927 21:07:34.138403 7845 raft_consensus.cc:2258] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:34.138669 7845 ts_tablet_manager.cc:1905] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:34.138804 7071 catalog_manager.cc:4841] TS 10f3f863734b4118ab156dda40c0811a (127.0.98.2:40637): tablet 4f9b00fdf91045689670edf6900d84a7 (table test-table [id=13715f8c376c48509d8e5bed8caa26a2]) successfully deleted
I20260927 21:07:34.140025 7845 ts_tablet_manager.cc:1918] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:34.140074 7845 log.cc:1196] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/4f9b00fdf91045689670edf6900d84a7
I20260927 21:07:34.140311 7845 ts_tablet_manager.cc:1939] T 4f9b00fdf91045689670edf6900d84a7 P c4fdc8ad18fa48d7997b4d3da4b31a20: Deleting consensus metadata
I20260927 21:07:34.140611 7068 catalog_manager.cc:4841] TS c4fdc8ad18fa48d7997b4d3da4b31a20 (127.0.98.3:41875): tablet 4f9b00fdf91045689670edf6900d84a7 (table test-table [id=13715f8c376c48509d8e5bed8caa26a2]) successfully deleted
I20260927 21:07:34.142048 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:34.141914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35131 (local address 127.0.0.1:33992)
0927 21:07:34.141976 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:34.142012 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:07:34.142174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 7458
W20260927 21:07:34.142194 7069 catalog_manager.cc:4584] TS ecf43902fdf141bc8c06e65a5cf881d7 (127.0.98.1:35131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f9b00fdf91045689670edf6900d84a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35131: connect: Connection refused (error 111)
I20260927 21:07:34.147347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 7594
I20260927 21:07:34.153343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 6950
I20260927 21:07:34.158018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 7138
2026-09-27T21:07:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4922 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:34 dist-test-slave-8bgs krb5kdc[7849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:34 dist-test-slave-8bgs krb5kdc[7849](info): set up 2 sockets
Sep 27 21:07:34 dist-test-slave-8bgs krb5kdc[7849](info): commencing operation
krb5kdc: starting...
W20260927 21:07:35.175586 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 21:07:35.283409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:35.361205 7862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:35.361397 7862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:35.361418 7862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:35.362483 7862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:35.362515 7862 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:07:35.362528 7862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:35.362541 7862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:35.362553 7862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:35.362565 7862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:35.363924 7862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:35.364096 7862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:35.364256 7862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:35.368749 7870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:35.368749 7868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:35.368789 7867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:35.369077 7862 server_base.cc:1034] running on GCE node
I20260927 21:07:35.369316 7862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:35.369571 7862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:35.370704 7862 hybrid_clock.cc:648] HybridClock initialized: now 1790543255370693 us; error 26 us; skew 500 ppm
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:35.373049 7862 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 21:07:35.373489 7862 webserver.cc:460] Webserver started at http://127.0.98.62:35791/ using document root <none> and password file <none>
I20260927 21:07:35.373703 7862 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:35.373767 7862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:35.373912 7862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:35.374895 7862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "8d589e2c89ce4a739f77fac4d3a0936f"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "1cf0af9ae992319f19074dbe7f2708a3"
server_key_iv: "37559f6052d0ee193cf0861565a3567c"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.375270 7862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d589e2c89ce4a739f77fac4d3a0936f"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "1cf0af9ae992319f19074dbe7f2708a3"
server_key_iv: "37559f6052d0ee193cf0861565a3567c"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.376537 7862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:35.377177 7877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.377367 7862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:35.377460 7862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "8d589e2c89ce4a739f77fac4d3a0936f"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "1cf0af9ae992319f19074dbe7f2708a3"
server_key_iv: "37559f6052d0ee193cf0861565a3567c"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.377540 7862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:35.404913 7862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:35.406716 7862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:35.406874 7862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:35.411255 7862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35137
I20260927 21:07:35.411294 7939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35137 every 8 connection(s)
I20260927 21:07:35.411605 7862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:35.412166 7940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:35.414513 7940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Bootstrap starting.
I20260927 21:07:35.415167 7940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:35.415474 7940 log.cc:824] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:35.416109 7940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: No bootstrap required, opened a new log
I20260927 21:07:35.417851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 7862
I20260927 21:07:35.417963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:35.418231 392 external_mini_cluster.cc:1428] Setting key 36da85b0c3b81bb5332d6794550d2289
I20260927 21:07:35.418622 7940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260927 21:07:35.418828 7940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:35.418907 7940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d589e2c89ce4a739f77fac4d3a0936f, State: Initialized, Role: FOLLOWER
I20260927 21:07:35.419062 7940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260927 21:07:35.419176 7940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:35.419220 7940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:35.419266 7940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:35.419838 7940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260927 21:07:35.419950 7940 leader_election.cc:302] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d589e2c89ce4a739f77fac4d3a0936f; no voters:
I20260927 21:07:35.420106 7940 leader_election.cc:288] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:35.420183 7945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:35.420336 7940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:35.420368 7945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [term 1 LEADER]: Becoming Leader. State: Replica: 8d589e2c89ce4a739f77fac4d3a0936f, State: Running, Role: LEADER
I20260927 21:07:35.420600 7945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } }
I20260927 21:07:35.421005 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d589e2c89ce4a739f77fac4d3a0936f. Latest consensus state: current_term: 1 leader_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } }
I20260927 21:07:35.421087 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:35.421489 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d589e2c89ce4a739f77fac4d3a0936f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35137 } } }
I20260927 21:07:35.421550 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:35.421881 7954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:35.422406 7954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:35.423866 7954 catalog_manager.cc:1347] Generated new cluster ID: 27f62f286a00412b85a4dfb8c20329b8
I20260927 21:07:35.423918 7954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:35.429296 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.418751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.98.62:35137)
0927 21:07:35.418945 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.418955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.418987 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:07:35.419149 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.419156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.419214 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.419366 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.420004 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.421520 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.422425 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.422883 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.423006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.425751 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.425754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.425754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.425765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.427685 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.428090 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.428093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.428093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.428205 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.428461 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.428462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.428462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.428727 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.428868 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.428997 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.429114 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260927 21:07:35.440622 7954 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:35.441151 7954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:35.446904 7954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Generated new TSK 0
I20260927 21:07:35.447084 7954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 21:07:35.474128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:35.582456 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:35.582705 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:35.582743 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:35.583958 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:35.584023 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:35.584059 7966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:35.584117 7966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:35.585600 7966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:35.585847 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:35.586066 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:35.591053 7966 server_base.cc:1034] running on GCE node
W20260927 21:07:35.590991 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:35.591015 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:35.590992 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:35.591400 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:35.591646 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:35.592808 7966 hybrid_clock.cc:648] HybridClock initialized: now 1790543255592792 us; error 47 us; skew 500 ppm
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:35.595355 7966 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:07:35.595722 7966 webserver.cc:460] Webserver started at http://127.0.98.1:37161/ using document root <none> and password file <none>
I20260927 21:07:35.595902 7966 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:35.595940 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:35.596024 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:35.597028 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "60c524ed5fec4ae4931da965b63c9a9b"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "8b24959cc777f93fc24b967953966194"
server_key_iv: "c790123347e57a1585fc62977aeff6c0"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.597373 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "60c524ed5fec4ae4931da965b63c9a9b"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "8b24959cc777f93fc24b967953966194"
server_key_iv: "c790123347e57a1585fc62977aeff6c0"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.598625 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:07:35.599345 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.599543 7966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:35.599635 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "60c524ed5fec4ae4931da965b63c9a9b"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "8b24959cc777f93fc24b967953966194"
server_key_iv: "c790123347e57a1585fc62977aeff6c0"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.599705 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:35.614951 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:35.616693 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:35.616842 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:35.617293 7966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:35.617331 7966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.617352 7966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:35.617394 7966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.624277 7966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34825
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:35.624333 8096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34825 every 8 connection(s)
I20260927 21:07:35.624637 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:35.627170 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.617755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37797 (local address 127.0.98.62:35137)
0927 21:07:35.617821 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.617824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.617982 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:07:35.618476 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.618478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.618490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.618547 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.619520 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.620217 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.621919 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.622304 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.622373 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.624815 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.624816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.624817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.624824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.626039 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.626395 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.626395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.626396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.626441 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.626635 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.626635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.626636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.626724 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.626770 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.626993 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.627052 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:07:35.627383 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.617736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:37797)
0927 21:07:35.617882 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.617917 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.618370 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.618666 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.618671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.619051 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:35.619367 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.619375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.620299 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.620301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.621810 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.621813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.622377 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.622378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.622505 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.622973 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:35.622983 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:35.624726 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:35.626111 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.626116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.626118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.626331 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.626495 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.626496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.626496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.626585 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.626776 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:35.626780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:35.626879 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.626973 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.627080 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260927 21:07:35.627875 8097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260927 21:07:35.627978 8097 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:35.628147 8097 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260927 21:07:35.628679 7893 ts_manager.cc:194] Registered new tserver with Master: 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825)
I20260927 21:07:35.629521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 7966
I20260927 21:07:35.629626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:35.629611 7893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37797
I20260927 21:07:35.629858 392 external_mini_cluster.cc:1428] Setting key a10ebfb6ed5dd315e861bc5379bc4bbe
I20260927 21:07:35.634656 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.628859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37951 (local address 127.0.98.62:35137)
0927 21:07:35.628898 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.628900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.629032 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:07:35.629090 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.629091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.629099 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.629144 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.629448 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.630151 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.630969 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.631298 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.631365 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.632202 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.632203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.632203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.632210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.633542 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.633915 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.633916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.633916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.633964 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.634165 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.634166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.634167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.634266 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.634337 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.634504 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.634578 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:35.634665 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.628817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.1:37951)
0927 21:07:35.628957 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.628967 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.629027 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.629149 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.629150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.629323 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:35.629378 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.629379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.630251 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.630252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.630889 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.630891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.631379 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.631379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.631423 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.631850 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:35.631855 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:35.632119 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:35.633674 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.633675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.633675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.633847 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.634043 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.634044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.634044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.634108 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.634330 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:35.634330 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:35.634368 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.634468 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.634562 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 21:07:35.653231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:35.732753 8106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:35.732934 8106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:35.732957 8106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:35.734009 8106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:35.734042 8106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:35.734061 8106 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:35.734078 8106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:35.735486 8106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:35.735651 8106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:35.735806 8106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:35.740448 8112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:35.740463 8114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:35.740548 8106 server_base.cc:1034] running on GCE node
W20260927 21:07:35.740489 8111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:35.740826 8106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:35.741070 8106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:35.742235 8106 hybrid_clock.cc:648] HybridClock initialized: now 1790543255742228 us; error 40 us; skew 500 ppm
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:35.744658 8106 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:07:35.745091 8106 webserver.cc:460] Webserver started at http://127.0.98.2:40257/ using document root <none> and password file <none>
I20260927 21:07:35.745304 8106 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:35.745368 8106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:35.745514 8106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:35.746457 8106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c3d6716f00f423e88dc3d608ed818a7"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "efecee6d181dd9e9d49f8539569f6733"
server_key_iv: "bef3e816e73347de8854f478d58ebff0"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.746788 8106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c3d6716f00f423e88dc3d608ed818a7"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "efecee6d181dd9e9d49f8539569f6733"
server_key_iv: "bef3e816e73347de8854f478d58ebff0"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.748071 8106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:35.748778 8121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.748947 8106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:35.749039 8106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "7c3d6716f00f423e88dc3d608ed818a7"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "efecee6d181dd9e9d49f8539569f6733"
server_key_iv: "bef3e816e73347de8854f478d58ebff0"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.749121 8106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:35.755236 8106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:35.757292 8106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:35.757431 8106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:35.757895 8106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:35.757948 8106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.757980 8106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:35.758008 8106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:35.764998 8106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43343
I20260927 21:07:35.765056 8236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43343 every 8 connection(s)
I20260927 21:07:35.765357 8106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:35.766561 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.758234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60067 (local address 127.0.98.62:35137)
0927 21:07:35.758292 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.758294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.758495 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:07:35.758979 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.758980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.758992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.759039 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.759634 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.760101 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.761103 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.761313 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.761395 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.763855 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.763857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.763857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.763865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.765385 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.765822 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.765823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.765823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.765901 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.766123 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.766123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.766123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.766214 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.766260 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.766450 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.766504 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:07:35.766778 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.758231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:60067)
0927 21:07:35.758376 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.758410 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.758879 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.759129 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.759132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.759339 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:35.759540 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.759544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.760186 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.760187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.760999 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.761001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.761404 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.761405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.761521 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.761988 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:35.761995 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:35.763754 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:35.765483 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.765488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.765490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.765763 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.765974 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.765975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.765975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.766077 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.766263 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:35.766266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:35.766364 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.766436 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.766538 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260927 21:07:35.767349 8237 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260927 21:07:35.767443 8237 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:35.767416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 8106
I20260927 21:07:35.767531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:35.767608 8237 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260927 21:07:35.767738 392 external_mini_cluster.cc:1428] Setting key c5c6c4473237f3c3feb5af137cb54d19
I20260927 21:07:35.767931 7894 ts_manager.cc:194] Registered new tserver with Master: 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343)
I20260927 21:07:35.768384 7894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60067
I20260927 21:07:35.775224 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.768351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.2:58007)
0927 21:07:35.769550 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.769559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.769609 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.769742 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.769743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.769880 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:35.769923 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.769924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.770762 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.770763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.771339 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.771340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.771813 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.771814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.771861 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.772254 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:35.772258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:35.772517 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:35.774211 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.774212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.774213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.774369 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.774567 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.774568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.774568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.774618 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.774930 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:35.774931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:35.774962 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.775010 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.775076 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":65,"threads_started":1}
I20260927 21:07:35.775281 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.768399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58007 (local address 127.0.98.62:35137)
0927 21:07:35.768433 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.768435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.769647 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 21:07:35.769672 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.769673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.769683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.769740 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.769982 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.770676 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.771412 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.771740 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.771816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.772602 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.772603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.772604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.772612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.774137 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.774442 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.774443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.774443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.774501 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.774689 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.774690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.774690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.774866 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.774935 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.775038 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.775151 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 21:07:35.792209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35137
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:35.870208 8246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:35.870388 8246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:35.870412 8246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:35.871475 8246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:35.871510 8246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:35.871531 8246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:07:35.871547 8246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:35.872886 8246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:35.873049 8246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:35.873198 8246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:35.877677 8254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:35.877830 8251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:35.877820 8246 server_base.cc:1034] running on GCE node
W20260927 21:07:35.877748 8252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:35.878152 8246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:35.878398 8246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:35.879549 8246 hybrid_clock.cc:648] HybridClock initialized: now 1790543255879537 us; error 25 us; skew 500 ppm
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:35.881791 8246 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:07:35.882211 8246 webserver.cc:460] Webserver started at http://127.0.98.3:39997/ using document root <none> and password file <none>
I20260927 21:07:35.882443 8246 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:35.882505 8246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:35.882658 8246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:35.883761 8246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "6290af3ade044bb18f19ee749ebe5c8a"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "a9f838acc200da69753d15324139204a"
server_key_iv: "867c996a489b839f50c414bffbdfd12f"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.884107 8246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6290af3ade044bb18f19ee749ebe5c8a"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "a9f838acc200da69753d15324139204a"
server_key_iv: "867c996a489b839f50c414bffbdfd12f"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.885332 8246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:35.886026 8261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.886190 8246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:35.886300 8246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "6290af3ade044bb18f19ee749ebe5c8a"
format_stamp: "Formatted at 2026-09-27 21:07:35 on dist-test-slave-8bgs"
server_key: "a9f838acc200da69753d15324139204a"
server_key_iv: "867c996a489b839f50c414bffbdfd12f"
server_key_version: "encryptionkey@0"
I20260927 21:07:35.886377 8246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:35.907624 8246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:35.909912 8246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:35.910018 8246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:35.910418 8246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:35.910449 8246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.910473 8246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:35.910519 8246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:35.917388 8246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36431
I20260927 21:07:35.917459 8376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36431 every 8 connection(s)
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:35.917761 8246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:35.920650 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.910923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55709 (local address 127.0.98.62:35137)
0927 21:07:35.910962 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.910964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.911259 (+ 295us) server_negotiation.cc:407] Connection header received
0927 21:07:35.911780 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.911782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.911795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.911845 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.912666 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.913511 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.915139 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.915469 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.915543 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.918260 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.918261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.918262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.918287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.919473 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.919851 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.919852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.919853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.919912 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.920124 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.920124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.920125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.920221 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.920271 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.920489 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.920582 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:35.920771 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.910917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:55709)
0927 21:07:35.911064 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.911143 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.911665 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.911909 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.911913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.912262 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:35.912546 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.912553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.913612 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.913613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.915006 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.915009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.915562 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.915563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.915725 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.916327 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:35.916338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:35.918161 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:35.919537 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.919543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.919544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.919749 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.919968 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.919971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.919971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.920070 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.920278 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:35.920282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:35.920388 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.920466 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.920577 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260927 21:07:35.921404 8377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35137
I20260927 21:07:35.921487 8377 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:35.921660 8377 heartbeater.cc:511] Master 127.0.98.62:35137 requested a full tablet report, sending...
I20260927 21:07:35.922092 7893 ts_manager.cc:194] Registered new tserver with Master: 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:35.922441 7893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55709
I20260927 21:07:35.926445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 8246
I20260927 21:07:35.926606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:35.926859 392 external_mini_cluster.cc:1428] Setting key 83d21286e82af0435f173f186b130a60
I20260927 21:07:35.927366 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.922509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34917 (local address 127.0.98.62:35137)
0927 21:07:35.922587 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.922589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.922663 (+ 74us) server_negotiation.cc:407] Connection header received
0927 21:07:35.922727 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.922727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.922735 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.922779 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.923133 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.923587 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.924202 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.924365 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.924410 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.925156 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.925157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.925157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.925162 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.926207 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.926525 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.926526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.926526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.926592 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.926810 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.926811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.926812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.926938 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.927004 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.927185 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.927260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:07:35.927378 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.922454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.98.3:34917)
0927 21:07:35.922597 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.922617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.922675 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.922782 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.922783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.922974 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:35.923026 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.923027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.923669 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.923669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.924136 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.924137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.924424 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.924425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.924470 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.924829 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:35.924833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:35.925084 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:35.926273 (+ 1189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.926274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.926274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.926429 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.926652 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:35.926652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:35.926653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:35.926740 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:35.927005 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:35.927006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:35.927054 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.927137 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.927235 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260927 21:07:35.928092 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:07:35.948416 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:35.943327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.0.98.62:35137)
0927 21:07:35.943399 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.943401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.943426 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:07:35.943485 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.943486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.943494 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.943544 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.944061 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.944534 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.945232 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.945403 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.945450 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.948304 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34818: BlockingRecv error: recv got EOF from 127.0.0.1:34818 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 21:07:35 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:35.955539 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.949144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.0.98.62:35137)
0927 21:07:35.949200 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.949202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.949258 (+ 56us) server_negotiation.cc:407] Connection header received
0927 21:07:35.949330 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.949331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.949342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.949414 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:35.949865 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.950319 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.951000 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.951236 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.951305 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.953284 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:35.953285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:35.953286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:35.953293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:35.954413 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.954743 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.954744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.954746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.954797 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:35.954988 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:35.954989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:35.954989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:35.955150 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:35.955257 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.955368 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.955439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:07:35.956959 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:35.958410 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:35.964927 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.960867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.0.98.1:34825)
0927 21:07:35.961032 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.961036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.961387 (+ 351us) server_negotiation.cc:407] Connection header received
0927 21:07:35.961546 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.961549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.961566 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.961639 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:35.961840 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.962505 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964059 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.964533 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964623 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.964701 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.964799 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.964838 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260927 21:07:35.964918 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.960750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.0.1:54052)
0927 21:07:35.961304 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.961314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.961390 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.961708 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.961709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.961711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:35.961751 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.961752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.962600 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.962601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.963959 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.963960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.964623 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.964695 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.964710 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.964785 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.964833 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":48,"thread_start_us":72,"threads_started":1}
I20260927 21:07:35.965006 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.960546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36431 (local address 127.0.0.1:50968)
0927 21:07:35.960694 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.960736 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.960796 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.961246 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.961249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.961251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:35.961344 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.961347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.962664 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.962665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.963958 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.963960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.964816 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.964879 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.964883 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.964911 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.964939 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":93,"thread_start_us":58,"threads_started":1}
I20260927 21:07:35.965114 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.960882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.0.98.2:43343)
0927 21:07:35.961055 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.961060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.961071 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:07:35.961111 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.961114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.961130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.961230 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:35.961437 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.962568 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.963872 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.964592 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964663 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.964735 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.964969 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.965007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260927 21:07:35.965220 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.960564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43343 (local address 127.0.0.1:53342)
0927 21:07:35.960869 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:35.960878 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:35.960913 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:35.961246 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:35.961249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:35.961254 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:35.961344 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.961346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.962653 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.962654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.963790 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:35.963792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.964769 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:35.964835 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.964839 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:35.964885 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:07:35.964928 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":118,"thread_start_us":80,"threads_started":1}
I20260927 21:07:35.965348 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:35.960680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.0.98.3:36431)
0927 21:07:35.961055 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:35.961060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:35.961071 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:07:35.961124 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:35.961126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:35.961139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:35.961230 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:35.961478 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.962543 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.964042 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:35.964728 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:35.965106 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:35.965191 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:35.965216 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 21:07:35.965261 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":104,"threads_started":1}
I20260927 21:07:35.965982 8311 tablet_service.cc:1467] Processing CreateTablet for tablet b23b253221a94d2e904f34f69f69ebd4 (DEFAULT_TABLE table=test-table [id=3e84f2f55d7a4527aca9cc9602859e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:35.965978 8031 tablet_service.cc:1467] Processing CreateTablet for tablet b23b253221a94d2e904f34f69f69ebd4 (DEFAULT_TABLE table=test-table [id=3e84f2f55d7a4527aca9cc9602859e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:35.965978 8171 tablet_service.cc:1467] Processing CreateTablet for tablet b23b253221a94d2e904f34f69f69ebd4 (DEFAULT_TABLE table=test-table [id=3e84f2f55d7a4527aca9cc9602859e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:35.966274 8311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23b253221a94d2e904f34f69f69ebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:35.966310 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23b253221a94d2e904f34f69f69ebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:35.966310 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23b253221a94d2e904f34f69f69ebd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:35.969244 8405 tablet_bootstrap.cc:492] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Bootstrap starting.
I20260927 21:07:35.969807 8405 tablet_bootstrap.cc:654] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:35.970116 8406 tablet_bootstrap.cc:492] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Bootstrap starting.
I20260927 21:07:35.970153 8405 log.cc:824] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:35.970290 8407 tablet_bootstrap.cc:492] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Bootstrap starting.
I20260927 21:07:35.970854 8407 tablet_bootstrap.cc:654] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:35.970916 8405 tablet_bootstrap.cc:492] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: No bootstrap required, opened a new log
I20260927 21:07:35.970971 8405 ts_tablet_manager.cc:1397] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:35.971235 8407 log.cc:824] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:35.971652 8406 tablet_bootstrap.cc:654] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:35.971956 8407 tablet_bootstrap.cc:492] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: No bootstrap required, opened a new log
I20260927 21:07:35.971971 8406 log.cc:824] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:35.972075 8407 ts_tablet_manager.cc:1397] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:35.972656 8406 tablet_bootstrap.cc:492] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: No bootstrap required, opened a new log
I20260927 21:07:35.972733 8406 ts_tablet_manager.cc:1397] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:07:35.973034 8405 raft_consensus.cc:348] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:35.973196 8405 raft_consensus.cc:374] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:35.973232 8405 raft_consensus.cc:729] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Initialized, Role: FOLLOWER
I20260927 21:07:35.973374 8405 consensus_queue.cc:260] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:35.973647 8405 ts_tablet_manager.cc:1428] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:07:35.973773 8377 heartbeater.cc:503] Master 127.0.98.62:35137 was elected leader, sending a full tablet report...
I20260927 21:07:35.973985 8407 raft_consensus.cc:348] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:35.974185 8407 raft_consensus.cc:374] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:35.974261 8407 raft_consensus.cc:729] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Initialized, Role: FOLLOWER
I20260927 21:07:35.974399 8407 consensus_queue.cc:260] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:35.974665 8407 ts_tablet_manager.cc:1428] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:07:35.974684 8237 heartbeater.cc:503] Master 127.0.98.62:35137 was elected leader, sending a full tablet report...
I20260927 21:07:35.975055 8406 raft_consensus.cc:348] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:35.975235 8406 raft_consensus.cc:374] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:35.975283 8406 raft_consensus.cc:729] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c524ed5fec4ae4931da965b63c9a9b, State: Initialized, Role: FOLLOWER
I20260927 21:07:35.975402 8406 consensus_queue.cc:260] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:35.975654 8406 ts_tablet_manager.cc:1428] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:07:35.975697 8097 heartbeater.cc:503] Master 127.0.98.62:35137 was elected leader, sending a full tablet report...
W20260927 21:07:36.015862 8238 tablet.cc:2369] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:07:36.125365 8098 tablet.cc:2369] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:07:36.168269 8378 tablet.cc:2369] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:36.224342 8411 raft_consensus.cc:482] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:36.224457 8411 raft_consensus.cc:504] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:36.224879 8411 leader_election.cc:288] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343), 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825)
I20260927 21:07:36.227696 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.225186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57727 (local address 127.0.98.2:43343)
0927 21:07:36.225230 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.225232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.225240 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:36.225268 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.225269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.225279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.225350 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:36.225499 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.225954 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.226895 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.227322 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.227402 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.227412 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.227599 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.227634 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:07:36.227713 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.225065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43343 (local address 127.0.98.3:57727)
0927 21:07:36.225122 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.225134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.225185 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.225354 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.225355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.225356 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:36.225422 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.225423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.226036 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.226037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.226799 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.226801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.227411 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.227412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.227500 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.227503 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.227573 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.227629 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:07:36.227789 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.225196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37971 (local address 127.0.98.1:34825)
0927 21:07:36.225241 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.225243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.225329 (+ 86us) server_negotiation.cc:407] Connection header received
0927 21:07:36.225398 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.225399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.225415 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.225487 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:36.225656 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.226088 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.226996 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.227410 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.227493 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.227502 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.227671 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.227703 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:36.227843 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.225093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.98.3:37971)
0927 21:07:36.225254 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.225263 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.225344 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.225493 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.225493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.225495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:36.225572 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.225573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.226178 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.226179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.226904 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.226906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.227493 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.227494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.227591 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.227595 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.227663 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.227701 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260927 21:07:36.228163 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23b253221a94d2e904f34f69f69ebd4" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b" is_pre_election: true
I20260927 21:07:36.228258 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23b253221a94d2e904f34f69f69ebd4" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3d6716f00f423e88dc3d608ed818a7" is_pre_election: true
I20260927 21:07:36.228351 8051 raft_consensus.cc:2454] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 0.
I20260927 21:07:36.228474 8191 raft_consensus.cc:2454] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 0.
I20260927 21:07:36.228597 8265 leader_election.cc:302] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 6290af3ade044bb18f19ee749ebe5c8a; no voters:
I20260927 21:07:36.228725 8411 raft_consensus.cc:2781] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:36.228771 8411 raft_consensus.cc:482] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:36.228803 8411 raft_consensus.cc:3037] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.229732 8411 raft_consensus.cc:504] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:36.229869 8411 leader_election.cc:288] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 election: Requested vote from peers 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343), 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825)
I20260927 21:07:36.230016 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23b253221a94d2e904f34f69f69ebd4" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3d6716f00f423e88dc3d608ed818a7"
I20260927 21:07:36.230039 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23b253221a94d2e904f34f69f69ebd4" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b"
I20260927 21:07:36.230098 8191 raft_consensus.cc:3037] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.230100 8051 raft_consensus.cc:3037] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.230919 8191 raft_consensus.cc:2454] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 1.
I20260927 21:07:36.230919 8051 raft_consensus.cc:2454] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 1.
I20260927 21:07:36.231139 8262 leader_election.cc:302] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6290af3ade044bb18f19ee749ebe5c8a, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.231251 8411 raft_consensus.cc:2781] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:36.231426 8411 raft_consensus.cc:686] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 LEADER]: Becoming Leader. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Running, Role: LEADER
I20260927 21:07:36.231527 8411 consensus_queue.cc:237] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } }
I20260927 21:07:36.232335 7893 catalog_manager.cc:5495] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6290af3ade044bb18f19ee749ebe5c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:36.245158 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.242814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.3:36431)
0927 21:07:36.242871 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.242872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.242881 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:36.242947 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.242948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.242972 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.243055 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.243220 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.243676 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.244406 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.244594 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.244690 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.244856 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.244949 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.245036 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.245054 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.245082 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:07:36.246730 8191 raft_consensus.cc:1261] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Refusing update from remote peer 6290af3ade044bb18f19ee749ebe5c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.246754 8051 raft_consensus.cc:1261] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Refusing update from remote peer 6290af3ade044bb18f19ee749ebe5c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.247035 8411 consensus_queue.cc:1035] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.247174 8415 consensus_queue.cc:1035] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.249735 8419 mvcc.cc:205] Tried to move back new op lower bound from 7334065177584259072 to 7334065177524961280. Current Snapshot: MvccSnapshot[applied={T|T < 7334065177584259072}]
I20260927 21:07:36.249969 8420 mvcc.cc:205] Tried to move back new op lower bound from 7334065177584259072 to 7334065177524961280. Current Snapshot: MvccSnapshot[applied={T|T < 7334065177584259072}]
I20260927 21:07:36.251528 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.251617 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.252281 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Sending DeleteTablet for 3 replicas of tablet b23b253221a94d2e904f34f69f69ebd4
I20260927 21:07:36.252616 8311 tablet_service.cc:1514] Processing DeleteTablet for tablet b23b253221a94d2e904f34f69f69ebd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50968
I20260927 21:07:36.252666 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet b23b253221a94d2e904f34f69f69ebd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53342
I20260927 21:07:36.252738 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:07:36.252867 8428 tablet_replica.cc:330] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: stopping tablet replica
W20260927 21:07:36.252888 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:36.252868 8031 tablet_service.cc:1514] Processing DeleteTablet for tablet b23b253221a94d2e904f34f69f69ebd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54052
I20260927 21:07:36.252954 8428 raft_consensus.cc:2229] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.253038 8428 raft_consensus.cc:2258] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.253118 8427 tablet_replica.cc:330] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: stopping tablet replica
I20260927 21:07:36.253235 8427 raft_consensus.cc:2229] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:36.253463 8429 tablet_replica.cc:330] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: stopping tablet replica
I20260927 21:07:36.253540 8428 ts_tablet_manager.cc:1905] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.253574 8429 raft_consensus.cc:2229] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.253701 8429 raft_consensus.cc:2258] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.254144 8429 ts_tablet_manager.cc:1905] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.254842 8031 tablet_service.cc:1467] Processing CreateTablet for tablet 39fdf47dfc2e468bb41c295230d473d8 (DEFAULT_TABLE table=test-table [id=7daea17eb71842aa849a2d37e17262a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.254951 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fdf47dfc2e468bb41c295230d473d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.255477 8428 ts_tablet_manager.cc:1918] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:36.255545 8428 log.cc:1196] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/b23b253221a94d2e904f34f69f69ebd4
I20260927 21:07:36.255792 8429 ts_tablet_manager.cc:1918] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:36.255813 8428 ts_tablet_manager.cc:1939] T b23b253221a94d2e904f34f69f69ebd4 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting consensus metadata
I20260927 21:07:36.255848 8429 log.cc:1196] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/b23b253221a94d2e904f34f69f69ebd4
I20260927 21:07:36.256084 8429 ts_tablet_manager.cc:1939] T b23b253221a94d2e904f34f69f69ebd4 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting consensus metadata
I20260927 21:07:36.256366 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 39fdf47dfc2e468bb41c295230d473d8 (DEFAULT_TABLE table=test-table [id=7daea17eb71842aa849a2d37e17262a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.256455 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fdf47dfc2e468bb41c295230d473d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.257189 8406 tablet_bootstrap.cc:492] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Bootstrap starting.
I20260927 21:07:36.257207 8407 tablet_bootstrap.cc:492] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Bootstrap starting.
I20260927 21:07:36.257469 8406 tablet_bootstrap.cc:654] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.257589 8407 tablet_bootstrap.cc:654] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.257876 7878 catalog_manager.cc:4841] TS 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343): tablet b23b253221a94d2e904f34f69f69ebd4 (table test-table [id=3e84f2f55d7a4527aca9cc9602859e17]) successfully deleted
I20260927 21:07:36.258095 7881 catalog_manager.cc:4841] TS 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825): tablet b23b253221a94d2e904f34f69f69ebd4 (table test-table [id=3e84f2f55d7a4527aca9cc9602859e17]) successfully deleted
I20260927 21:07:36.258569 8406 tablet_bootstrap.cc:492] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: No bootstrap required, opened a new log
I20260927 21:07:36.258618 8406 ts_tablet_manager.cc:1397] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:36.258915 8406 raft_consensus.cc:348] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.258991 8406 raft_consensus.cc:374] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.259014 8406 raft_consensus.cc:729] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c524ed5fec4ae4931da965b63c9a9b, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.259060 8406 consensus_queue.cc:260] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.253377 8427 raft_consensus.cc:2258] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.259188 8406 ts_tablet_manager.cc:1428] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:36.256302 8311 tablet_service.cc:1467] Processing CreateTablet for tablet 39fdf47dfc2e468bb41c295230d473d8 (DEFAULT_TABLE table=test-table [id=7daea17eb71842aa849a2d37e17262a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.259443 8311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fdf47dfc2e468bb41c295230d473d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.259555 8407 tablet_bootstrap.cc:492] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: No bootstrap required, opened a new log
I20260927 21:07:36.259608 8407 ts_tablet_manager.cc:1397] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:36.259733 8407 raft_consensus.cc:348] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.259788 8407 raft_consensus.cc:374] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.259809 8407 raft_consensus.cc:729] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.259858 8407 consensus_queue.cc:260] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.259943 8407 ts_tablet_manager.cc:1428] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.260537 8405 tablet_bootstrap.cc:492] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Bootstrap starting.
I20260927 21:07:36.260974 8405 tablet_bootstrap.cc:654] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.261015 8427 ts_tablet_manager.cc:1905] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.261648 8405 tablet_bootstrap.cc:492] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: No bootstrap required, opened a new log
I20260927 21:07:36.261688 8405 ts_tablet_manager.cc:1397] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:36.261816 8405 raft_consensus.cc:348] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.261891 8405 raft_consensus.cc:374] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.261919 8405 raft_consensus.cc:729] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.261973 8405 consensus_queue.cc:260] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.262058 8405 ts_tablet_manager.cc:1428] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.262579 8427 ts_tablet_manager.cc:1918] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:36.262634 8427 log.cc:1196] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/b23b253221a94d2e904f34f69f69ebd4
I20260927 21:07:36.262809 8427 ts_tablet_manager.cc:1939] T b23b253221a94d2e904f34f69f69ebd4 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting consensus metadata
I20260927 21:07:36.263306 7881 catalog_manager.cc:4841] TS 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431): tablet b23b253221a94d2e904f34f69f69ebd4 (table test-table [id=3e84f2f55d7a4527aca9cc9602859e17]) successfully deleted
I20260927 21:07:36.384649 8413 raft_consensus.cc:482] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:36.384745 8413 raft_consensus.cc:504] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.385105 8413 leader_election.cc:288] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343), 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:36.388069 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.385353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34553 (local address 127.0.98.2:43343)
0927 21:07:36.385412 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.385415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.385570 (+ 155us) server_negotiation.cc:407] Connection header received
0927 21:07:36.385630 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.385631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.385640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.385695 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:36.385865 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.386436 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.387340 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.387695 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.387749 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.387757 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.387949 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.387995 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:07:36.388059 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.385279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43343 (local address 127.0.98.1:34553)
0927 21:07:36.385479 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.385489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.385556 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.385705 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.385705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.385707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:36.385791 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.385793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.386523 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.386524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.387262 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.387264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.387792 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.387794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.387878 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.387882 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.387922 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.387981 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260927 21:07:36.388231 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.385264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36431 (local address 127.0.98.1:56953)
0927 21:07:36.385403 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.385414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.385484 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.385632 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.385633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.385635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:36.385716 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.385719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.386353 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.386354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.387267 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.387269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.387833 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.387834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.387884 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.387887 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.388089 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.388135 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260927 21:07:36.388265 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.385342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56953 (local address 127.0.98.3:36431)
0927 21:07:36.385391 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.385393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.385489 (+ 96us) server_negotiation.cc:407] Connection header received
0927 21:07:36.385527 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.385528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.385539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.385608 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:36.385792 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.386280 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.387376 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.387744 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.387817 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.387827 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.388116 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.388166 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:07:36.388393 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fdf47dfc2e468bb41c295230d473d8" candidate_uuid: "60c524ed5fec4ae4931da965b63c9a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3d6716f00f423e88dc3d608ed818a7" is_pre_election: true
I20260927 21:07:36.388497 8191 raft_consensus.cc:2454] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c524ed5fec4ae4931da965b63c9a9b in term 0.
I20260927 21:07:36.388450 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fdf47dfc2e468bb41c295230d473d8" candidate_uuid: "60c524ed5fec4ae4931da965b63c9a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290af3ade044bb18f19ee749ebe5c8a" is_pre_election: true
I20260927 21:07:36.388628 8331 raft_consensus.cc:2454] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c524ed5fec4ae4931da965b63c9a9b in term 0.
I20260927 21:07:36.388682 7982 leader_election.cc:302] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.388806 8413 raft_consensus.cc:2781] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:36.388870 8413 raft_consensus.cc:482] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:36.388899 8413 raft_consensus.cc:3037] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.389591 8413 raft_consensus.cc:504] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.389768 8413 leader_election.cc:288] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [CANDIDATE]: Term 1 election: Requested vote from peers 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343), 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:36.389930 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fdf47dfc2e468bb41c295230d473d8" candidate_uuid: "60c524ed5fec4ae4931da965b63c9a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3d6716f00f423e88dc3d608ed818a7"
I20260927 21:07:36.389955 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fdf47dfc2e468bb41c295230d473d8" candidate_uuid: "60c524ed5fec4ae4931da965b63c9a9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290af3ade044bb18f19ee749ebe5c8a"
I20260927 21:07:36.390007 8191 raft_consensus.cc:3037] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.390030 8331 raft_consensus.cc:3037] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.390632 8191 raft_consensus.cc:2454] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c524ed5fec4ae4931da965b63c9a9b in term 1.
I20260927 21:07:36.390717 8331 raft_consensus.cc:2454] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c524ed5fec4ae4931da965b63c9a9b in term 1.
I20260927 21:07:36.390793 7982 leader_election.cc:302] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.390872 8413 raft_consensus.cc:2781] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:36.391028 8413 raft_consensus.cc:686] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 LEADER]: Becoming Leader. State: Replica: 60c524ed5fec4ae4931da965b63c9a9b, State: Running, Role: LEADER
I20260927 21:07:36.391136 8413 consensus_queue.cc:237] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.391793 7894 catalog_manager.cc:5495] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "60c524ed5fec4ae4931da965b63c9a9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:07:36 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543255, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:36.432768 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.419682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35137 (local address 127.0.0.1:39436)
0927 21:07:36.419847 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.419857 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.419918 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.420057 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.420059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.420365 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:36.420407 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.420409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.421067 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.421068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.421651 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.421653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.421969 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.421970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.422070 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.429141 (+ 7071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:36.429146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:36.430482 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:36.431878 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:36.431884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:36.431885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:36.432079 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:36.432246 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:36.432246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:36.432247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:36.432307 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:36.432501 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:36.432505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:36.432544 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.432597 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.432680 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260927 21:07:36.432816 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.419727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.0.98.62:35137)
0927 21:07:36.419785 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.419787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.419918 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:07:36.419972 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.419972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.419981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.420021 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:36.420473 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.421006 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.421715 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.421887 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.421939 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.430561 (+ 8622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:36.430562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:36.430562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:36.430568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:36.431809 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:36.432140 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:36.432141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:36.432141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:36.432187 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:36.432357 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:36.432357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:36.432357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:36.432450 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:36.432498 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.432629 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.432725 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":35}
I20260927 21:07:36.434643 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:07:36.434841 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:36.436270 8311 tablet_service.cc:1467] Processing CreateTablet for tablet 9efcf0bd3c5449899ee379cc2b07d58f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dec8e9b0515848f383cf2fa7892e4040]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:36.436290 8031 tablet_service.cc:1467] Processing CreateTablet for tablet 9efcf0bd3c5449899ee379cc2b07d58f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dec8e9b0515848f383cf2fa7892e4040]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:36.436319 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 9efcf0bd3c5449899ee379cc2b07d58f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dec8e9b0515848f383cf2fa7892e4040]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:07:36.436436 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efcf0bd3c5449899ee379cc2b07d58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.436435 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efcf0bd3c5449899ee379cc2b07d58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.436437 8311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9efcf0bd3c5449899ee379cc2b07d58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.437562 8405 tablet_bootstrap.cc:492] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: Bootstrap starting.
I20260927 21:07:36.437831 8406 tablet_bootstrap.cc:492] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: Bootstrap starting.
I20260927 21:07:36.437842 8407 tablet_bootstrap.cc:492] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: Bootstrap starting.
I20260927 21:07:36.437949 8405 tablet_bootstrap.cc:654] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.438176 8407 tablet_bootstrap.cc:654] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.438248 8406 tablet_bootstrap.cc:654] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.438575 8405 tablet_bootstrap.cc:492] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: No bootstrap required, opened a new log
I20260927 21:07:36.438622 8405 ts_tablet_manager.cc:1397] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:36.438752 8407 tablet_bootstrap.cc:492] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: No bootstrap required, opened a new log
I20260927 21:07:36.438727 8405 raft_consensus.cc:348] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.438802 8407 ts_tablet_manager.cc:1397] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:36.438813 8405 raft_consensus.cc:374] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.438838 8405 raft_consensus.cc:729] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.438894 8405 consensus_queue.cc:260] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.438941 8407 raft_consensus.cc:348] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.438990 8405 ts_tablet_manager.cc:1428] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.439010 8407 raft_consensus.cc:374] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.439034 8407 raft_consensus.cc:729] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.439018 8411 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.439105 8407 consensus_queue.cc:260] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.439126 8411 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.439203 8407 ts_tablet_manager.cc:1428] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.439252 8412 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.439383 8412 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.440007 8406 tablet_bootstrap.cc:492] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: No bootstrap required, opened a new log
I20260927 21:07:36.440062 8406 ts_tablet_manager.cc:1397] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:36.440197 8406 raft_consensus.cc:348] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.440267 8406 raft_consensus.cc:374] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.440300 8406 raft_consensus.cc:729] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c524ed5fec4ae4931da965b63c9a9b, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.440351 8406 consensus_queue.cc:260] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.440435 8406 ts_tablet_manager.cc:1428] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.440485 8413 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.440583 8413 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.560647 8412 raft_consensus.cc:482] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:36.560757 8412 raft_consensus.cc:504] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.561103 8412 leader_election.cc:288] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825), 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:36.564076 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.561534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43977 (local address 127.0.98.3:36431)
0927 21:07:36.561577 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.561580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.561620 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:07:36.561704 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.561705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.561717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.561798 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:36.561948 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.562409 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.563400 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.563741 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.563816 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.563826 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.563964 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.563999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:36.564095 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.561462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36431 (local address 127.0.98.2:43977)
0927 21:07:36.561550 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.561571 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.561634 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.561799 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.561800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.561802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:36.561878 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.561879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.562497 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.562498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.563289 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.563291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.563815 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.563816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.563886 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.563889 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.563947 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.563985 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":17,"thread_start_us":50,"threads_started":1}
I20260927 21:07:36.564109 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.561282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.98.2:56753)
0927 21:07:36.561445 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.561459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.561560 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.561739 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.561739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.561741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:36.561819 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.561820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.562462 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.562463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.563333 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.563335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.563842 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.563842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.563915 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.563918 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.563976 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.564014 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":25,"thread_start_us":83,"threads_started":1}
I20260927 21:07:36.564091 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.561403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56753 (local address 127.0.98.1:34825)
0927 21:07:36.561465 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.561468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.561531 (+ 63us) server_negotiation.cc:407] Connection header received
0927 21:07:36.561639 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.561640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.561650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.561720 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:36.561887 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.562369 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.563424 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.563761 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.563841 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.563850 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.563993 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.564026 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:07:36.564338 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efcf0bd3c5449899ee379cc2b07d58f" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290af3ade044bb18f19ee749ebe5c8a" is_pre_election: true
I20260927 21:07:36.564327 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efcf0bd3c5449899ee379cc2b07d58f" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b" is_pre_election: true
I20260927 21:07:36.564415 8331 raft_consensus.cc:2454] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 0.
I20260927 21:07:36.564422 8051 raft_consensus.cc:2454] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 0.
I20260927 21:07:36.564571 8125 leader_election.cc:302] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.564697 8412 raft_consensus.cc:2781] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:36.564735 8412 raft_consensus.cc:482] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:36.564755 8412 raft_consensus.cc:3037] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.565402 8412 raft_consensus.cc:504] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.565533 8412 leader_election.cc:288] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 election: Requested vote from peers 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825), 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:36.565683 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efcf0bd3c5449899ee379cc2b07d58f" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b"
I20260927 21:07:36.565745 8051 raft_consensus.cc:3037] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.565691 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9efcf0bd3c5449899ee379cc2b07d58f" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290af3ade044bb18f19ee749ebe5c8a"
I20260927 21:07:36.565785 8331 raft_consensus.cc:3037] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.566566 8331 raft_consensus.cc:2454] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 1.
I20260927 21:07:36.566577 8051 raft_consensus.cc:2454] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 1.
I20260927 21:07:36.566736 8125 leader_election.cc:302] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.566821 8412 raft_consensus.cc:2781] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:36.566931 8412 raft_consensus.cc:686] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Running, Role: LEADER
I20260927 21:07:36.567003 8412 consensus_queue.cc:237] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.567360 8448 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: TxnStatusTablet state changed. Reason: New leader 7c3d6716f00f423e88dc3d608ed818a7. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.567441 8448 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:36.567641 8450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:07:36.567762 8450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:07:36.567749 7894 catalog_manager.cc:5495] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:36.583945 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.581570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.98.2:43343)
0927 21:07:36.581627 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.581628 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.581638 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:36.581724 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.581725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.581740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.581802 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.581924 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.582377 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.583168 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.583381 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.583453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.583639 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.583731 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.583812 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.583867 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.583883 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:07:36.583984 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.581512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43343 (local address 127.0.0.1:38914)
0927 21:07:36.581599 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.581610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.581658 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.581815 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.581816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.581817 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:36.581861 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.581863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.582462 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.582463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.583021 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.583022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.583466 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.583467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.583519 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.583532 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.583799 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.583807 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.583862 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.583909 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:07:36.585048 8331 raft_consensus.cc:1261] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Refusing update from remote peer 7c3d6716f00f423e88dc3d608ed818a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.585242 8051 raft_consensus.cc:1261] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Refusing update from remote peer 7c3d6716f00f423e88dc3d608ed818a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.585301 8412 consensus_queue.cc:1035] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.585425 8448 consensus_queue.cc:1035] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.585924 8411 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: TxnStatusTablet state changed. Reason: New leader 7c3d6716f00f423e88dc3d608ed818a7. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.586014 8411 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.585963 8413 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: TxnStatusTablet state changed. Reason: New leader 7c3d6716f00f423e88dc3d608ed818a7. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.586043 8413 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.586680 8412 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.586753 8412 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:36.587023 8448 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.587121 8448 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 7c3d6716f00f423e88dc3d608ed818a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:07:36.587446 8413 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.587527 8413 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 60c524ed5fec4ae4931da965b63c9a9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.588176 8418 mvcc.cc:205] Tried to move back new op lower bound from 7334065178970226688 to 7334065178899013632. Current Snapshot: MvccSnapshot[applied={T|T < 7334065178970226688 or (T in {7334065178970226688})}]
I20260927 21:07:36.588272 8411 tablet_replica.cc:439] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } } }
I20260927 21:07:36.588337 8411 tablet_replica.cc:442] T 9efcf0bd3c5449899ee379cc2b07d58f P 6290af3ade044bb18f19ee749ebe5c8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:07:36.591091 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.588445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.0.98.1:34825)
0927 21:07:36.588601 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.588603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.588611 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:36.588654 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.588654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.588673 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.588733 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.588896 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.589303 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.590360 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.590567 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.590617 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.590778 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.590854 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.590919 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.590992 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.591006 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260927 21:07:36.595829 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.593258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43343 (local address 127.0.98.1:39087)
0927 21:07:36.593381 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.593391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.593448 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.593569 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.593570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.593571 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:36.593614 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.593615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.594450 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.594450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.595025 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.595026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.595484 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.595484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.595530 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.595541 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.595709 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.595710 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.595742 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.595769 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
I20260927 21:07:36.595822 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.593326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39087 (local address 127.0.98.2:43343)
0927 21:07:36.593365 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.593367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.593449 (+ 82us) server_negotiation.cc:407] Connection header received
0927 21:07:36.593487 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.593488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.593496 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.593562 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.593673 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.594373 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.595120 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.595407 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.595468 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.595625 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.595654 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.595704 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.595745 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.595760 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:07:36.597894 8331 raft_consensus.cc:1261] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Refusing update from remote peer 60c524ed5fec4ae4931da965b63c9a9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.597922 8191 raft_consensus.cc:1261] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Refusing update from remote peer 60c524ed5fec4ae4931da965b63c9a9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.598135 8413 consensus_queue.cc:1035] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.598284 8435 consensus_queue.cc:1035] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.606901 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.603917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37619 (local address 127.0.98.1:34825)
0927 21:07:36.603955 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.603958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.604075 (+ 117us) server_negotiation.cc:407] Connection header received
0927 21:07:36.604142 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.604143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.604152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.604207 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.604467 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.605138 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.606257 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.606452 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.606481 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.606688 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.606717 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.606758 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.606819 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.606835 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:36.606932 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.603878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.98.2:37619)
0927 21:07:36.604015 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.604026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.604095 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.604302 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.604302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.604340 (+ 38us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:36.604404 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.604406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.605217 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.605218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.606146 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.606147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.606520 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.606520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.606563 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.606573 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.606769 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.606771 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.606807 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.606842 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260927 21:07:36.615978 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.616072 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.616655 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Sending DeleteTablet for 3 replicas of tablet 39fdf47dfc2e468bb41c295230d473d8
I20260927 21:07:36.616990 8031 tablet_service.cc:1514] Processing DeleteTablet for tablet 39fdf47dfc2e468bb41c295230d473d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54052
I20260927 21:07:36.617086 8311 tablet_service.cc:1514] Processing DeleteTablet for tablet 39fdf47dfc2e468bb41c295230d473d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50968
I20260927 21:07:36.617172 8429 tablet_replica.cc:330] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: stopping tablet replica
I20260927 21:07:36.617226 8429 raft_consensus.cc:2229] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:36.617300 8429 raft_consensus.cc:2258] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.617372 8427 tablet_replica.cc:330] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: stopping tablet replica
I20260927 21:07:36.617424 8427 raft_consensus.cc:2229] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.617465 8427 raft_consensus.cc:2258] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.617642 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet 39fdf47dfc2e468bb41c295230d473d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53342
I20260927 21:07:36.617722 8428 tablet_replica.cc:330] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: stopping tablet replica
I20260927 21:07:36.617767 8428 raft_consensus.cc:2229] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.617805 8428 raft_consensus.cc:2258] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.618136 8428 ts_tablet_manager.cc:1905] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.619328 8429 ts_tablet_manager.cc:1905] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.619778 8428 ts_tablet_manager.cc:1918] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:36.619827 8428 log.cc:1196] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/39fdf47dfc2e468bb41c295230d473d8
I20260927 21:07:36.620029 8428 ts_tablet_manager.cc:1939] T 39fdf47dfc2e468bb41c295230d473d8 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting consensus metadata
I20260927 21:07:36.620327 7878 catalog_manager.cc:4841] TS 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343): tablet 39fdf47dfc2e468bb41c295230d473d8 (table test-table [id=7daea17eb71842aa849a2d37e17262a6]) successfully deleted
I20260927 21:07:36.620916 8429 ts_tablet_manager.cc:1918] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:36.620963 8429 log.cc:1196] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/39fdf47dfc2e468bb41c295230d473d8
I20260927 21:07:36.621165 8429 ts_tablet_manager.cc:1939] T 39fdf47dfc2e468bb41c295230d473d8 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting consensus metadata
I20260927 21:07:36.621676 8427 ts_tablet_manager.cc:1905] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.623162 8427 ts_tablet_manager.cc:1918] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:36.623217 8427 log.cc:1196] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/39fdf47dfc2e468bb41c295230d473d8
I20260927 21:07:36.623435 8427 ts_tablet_manager.cc:1939] T 39fdf47dfc2e468bb41c295230d473d8 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting consensus metadata
I20260927 21:07:36.625674 7881 catalog_manager.cc:4841] TS 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825): tablet 39fdf47dfc2e468bb41c295230d473d8 (table test-table [id=7daea17eb71842aa849a2d37e17262a6]) successfully deleted
I20260927 21:07:36.626355 7881 catalog_manager.cc:4841] TS 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431): tablet 39fdf47dfc2e468bb41c295230d473d8 (table test-table [id=7daea17eb71842aa849a2d37e17262a6]) successfully deleted
Sep 27 21:07:36 dist-test-slave-8bgs krb5kdc[7849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:07:36 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:36 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:07:36.652556 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:36.645635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.0.98.62:35137)
0927 21:07:36.645687 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.645689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.645802 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:07:36.645844 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.645845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.645858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.645934 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:36.646646 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.647481 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.648591 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.648884 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.648956 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.652491 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39444: BlockingRecv error: recv got EOF from 127.0.0.1:39444 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 21:07:36 dist-test-slave-8bgs krb5kdc[7849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 21:07:36.661954 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.653553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.0.98.62:35137)
0927 21:07:36.653587 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.653589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.653684 (+ 95us) server_negotiation.cc:407] Connection header received
0927 21:07:36.653731 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.653731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.653741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.653798 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:36.654386 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.655192 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.656147 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.656435 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.656501 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.659019 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:36.659020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:36.659020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:36.659028 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:36.660651 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:36.661027 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:36.661029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:36.661029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:36.661092 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:36.661330 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:36.661331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:36.661332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:36.661466 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:36.661568 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.661689 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.661763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:36.663477 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:36.663659 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:36.665060 8171 tablet_service.cc:1467] Processing CreateTablet for tablet bb72a51935e54ca59616459d49beccc7 (DEFAULT_TABLE table=test-table [id=c8d0c6bfdb6a4d74b1707d2c806d4231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.665148 8031 tablet_service.cc:1467] Processing CreateTablet for tablet bb72a51935e54ca59616459d49beccc7 (DEFAULT_TABLE table=test-table [id=c8d0c6bfdb6a4d74b1707d2c806d4231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.665197 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb72a51935e54ca59616459d49beccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.665266 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb72a51935e54ca59616459d49beccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.666559 8406 tablet_bootstrap.cc:492] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Bootstrap starting.
I20260927 21:07:36.666576 8311 tablet_service.cc:1467] Processing CreateTablet for tablet bb72a51935e54ca59616459d49beccc7 (DEFAULT_TABLE table=test-table [id=c8d0c6bfdb6a4d74b1707d2c806d4231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.666679 8311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb72a51935e54ca59616459d49beccc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.667006 8406 tablet_bootstrap.cc:654] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.667351 8407 tablet_bootstrap.cc:492] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Bootstrap starting.
I20260927 21:07:36.667781 8407 tablet_bootstrap.cc:654] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.667941 8405 tablet_bootstrap.cc:492] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Bootstrap starting.
I20260927 21:07:36.668255 8405 tablet_bootstrap.cc:654] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.668690 8407 tablet_bootstrap.cc:492] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: No bootstrap required, opened a new log
I20260927 21:07:36.668740 8407 ts_tablet_manager.cc:1397] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:36.668895 8407 raft_consensus.cc:348] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.668975 8407 raft_consensus.cc:374] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.669003 8407 raft_consensus.cc:729] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.669046 8407 consensus_queue.cc:260] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.669135 8407 ts_tablet_manager.cc:1428] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:36.669288 8405 tablet_bootstrap.cc:492] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: No bootstrap required, opened a new log
I20260927 21:07:36.669332 8405 ts_tablet_manager.cc:1397] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:36.669418 8405 raft_consensus.cc:348] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.669469 8405 raft_consensus.cc:374] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.669486 8405 raft_consensus.cc:729] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.669523 8405 consensus_queue.cc:260] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.669561 8406 tablet_bootstrap.cc:492] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: No bootstrap required, opened a new log
I20260927 21:07:36.669592 8405 ts_tablet_manager.cc:1428] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.669615 8406 ts_tablet_manager.cc:1397] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:07:36.669773 8406 raft_consensus.cc:348] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.669831 8406 raft_consensus.cc:374] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.669849 8406 raft_consensus.cc:729] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c524ed5fec4ae4931da965b63c9a9b, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.669889 8406 consensus_queue.cc:260] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.669972 8406 ts_tablet_manager.cc:1428] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.845829 8412 raft_consensus.cc:482] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:36.845933 8412 raft_consensus.cc:504] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.846124 8412 leader_election.cc:288] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825), 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:36.846388 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb72a51935e54ca59616459d49beccc7" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b" is_pre_election: true
I20260927 21:07:36.846388 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb72a51935e54ca59616459d49beccc7" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290af3ade044bb18f19ee749ebe5c8a" is_pre_election: true
I20260927 21:07:36.846493 8331 raft_consensus.cc:2454] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 0.
I20260927 21:07:36.846493 8051 raft_consensus.cc:2454] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 0.
I20260927 21:07:36.846638 8125 leader_election.cc:302] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.846741 8412 raft_consensus.cc:2781] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:36.846791 8412 raft_consensus.cc:482] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:36.846819 8412 raft_consensus.cc:3037] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.847572 8412 raft_consensus.cc:504] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.847709 8412 leader_election.cc:288] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 election: Requested vote from peers 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825), 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431)
I20260927 21:07:36.847888 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb72a51935e54ca59616459d49beccc7" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b"
I20260927 21:07:36.847921 8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb72a51935e54ca59616459d49beccc7" candidate_uuid: "7c3d6716f00f423e88dc3d608ed818a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6290af3ade044bb18f19ee749ebe5c8a"
I20260927 21:07:36.847977 8331 raft_consensus.cc:3037] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.847980 8051 raft_consensus.cc:3037] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.848526 8051 raft_consensus.cc:2454] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 1.
I20260927 21:07:36.848594 8331 raft_consensus.cc:2454] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3d6716f00f423e88dc3d608ed818a7 in term 1.
I20260927 21:07:36.848747 8125 leader_election.cc:302] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.848905 8412 raft_consensus.cc:2781] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:36.848997 8412 raft_consensus.cc:686] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Running, Role: LEADER
I20260927 21:07:36.849050 8412 consensus_queue.cc:237] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.849602 7893 catalog_manager.cc:5495] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7c3d6716f00f423e88dc3d608ed818a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } health_report { overall_health: UNKNOWN } } }
I20260927 21:07:36.897596 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.895672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.0.98.2:43343)
0927 21:07:36.895732 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.895735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.895758 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:07:36.895778 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.895779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.895787 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.895823 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.895945 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.896389 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.897055 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.897234 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.897278 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.897394 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.897416 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.897449 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.897495 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.897508 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:07:36.898455 8331 raft_consensus.cc:1261] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Refusing update from remote peer 7c3d6716f00f423e88dc3d608ed818a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.898494 8051 raft_consensus.cc:1261] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Refusing update from remote peer 7c3d6716f00f423e88dc3d608ed818a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.898741 8460 consensus_queue.cc:1035] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.898867 8412 consensus_queue.cc:1035] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.901808 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.901918 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.902547 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Sending DeleteTablet for 3 replicas of tablet bb72a51935e54ca59616459d49beccc7
I20260927 21:07:36.902830 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet bb72a51935e54ca59616459d49beccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53342
I20260927 21:07:36.902842 8311 tablet_service.cc:1514] Processing DeleteTablet for tablet bb72a51935e54ca59616459d49beccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50968
I20260927 21:07:36.902925 8428 tablet_replica.cc:330] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: stopping tablet replica
I20260927 21:07:36.902976 8428 raft_consensus.cc:2229] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:36.903069 8428 raft_consensus.cc:2258] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.903167 8427 tablet_replica.cc:330] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: stopping tablet replica
I20260927 21:07:36.903254 8427 raft_consensus.cc:2229] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.903293 8427 raft_consensus.cc:2258] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.903414 8428 ts_tablet_manager.cc:1905] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.903697 7893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:36.903847 7893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:36.904886 8428 ts_tablet_manager.cc:1918] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:36.904946 8428 log.cc:1196] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/bb72a51935e54ca59616459d49beccc7
I20260927 21:07:36.905164 8428 ts_tablet_manager.cc:1939] T bb72a51935e54ca59616459d49beccc7 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting consensus metadata
I20260927 21:07:36.905402 8031 tablet_service.cc:1514] Processing DeleteTablet for tablet bb72a51935e54ca59616459d49beccc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54052
I20260927 21:07:36.905454 7878 catalog_manager.cc:4841] TS 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343): tablet bb72a51935e54ca59616459d49beccc7 (table test-table [id=c8d0c6bfdb6a4d74b1707d2c806d4231]) successfully deleted
I20260927 21:07:36.905498 8429 tablet_replica.cc:330] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: stopping tablet replica
I20260927 21:07:36.905552 8429 raft_consensus.cc:2229] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.905592 8429 raft_consensus.cc:2258] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.905905 8427 ts_tablet_manager.cc:1905] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.906165 8429 ts_tablet_manager.cc:1905] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.906474 8311 tablet_service.cc:1467] Processing CreateTablet for tablet 30705240040b438f832c987d23804405 (DEFAULT_TABLE table=test-table [id=d3fae82eedb34446b5034916b40a4d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.906569 8311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30705240040b438f832c987d23804405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.907536 8427 ts_tablet_manager.cc:1918] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:36.907590 8427 log.cc:1196] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/bb72a51935e54ca59616459d49beccc7
I20260927 21:07:36.907809 8427 ts_tablet_manager.cc:1939] T bb72a51935e54ca59616459d49beccc7 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting consensus metadata
I20260927 21:07:36.907835 8429 ts_tablet_manager.cc:1918] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:36.907878 8429 log.cc:1196] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/bb72a51935e54ca59616459d49beccc7
I20260927 21:07:36.908017 8031 tablet_service.cc:1467] Processing CreateTablet for tablet 30705240040b438f832c987d23804405 (DEFAULT_TABLE table=test-table [id=d3fae82eedb34446b5034916b40a4d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.908073 8429 ts_tablet_manager.cc:1939] T bb72a51935e54ca59616459d49beccc7 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting consensus metadata
I20260927 21:07:36.908123 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30705240040b438f832c987d23804405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.909216 8406 tablet_bootstrap.cc:492] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: Bootstrap starting.
I20260927 21:07:36.909267 7881 catalog_manager.cc:4841] TS 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825): tablet bb72a51935e54ca59616459d49beccc7 (table test-table [id=c8d0c6bfdb6a4d74b1707d2c806d4231]) successfully deleted
I20260927 21:07:36.909368 7881 catalog_manager.cc:4841] TS 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431): tablet bb72a51935e54ca59616459d49beccc7 (table test-table [id=c8d0c6bfdb6a4d74b1707d2c806d4231]) successfully deleted
I20260927 21:07:36.909512 8406 tablet_bootstrap.cc:654] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.909559 8405 tablet_bootstrap.cc:492] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Bootstrap starting.
I20260927 21:07:36.909798 8171 tablet_service.cc:1467] Processing CreateTablet for tablet 30705240040b438f832c987d23804405 (DEFAULT_TABLE table=test-table [id=d3fae82eedb34446b5034916b40a4d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:36.909857 8405 tablet_bootstrap.cc:654] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.909897 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30705240040b438f832c987d23804405. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:36.910379 8405 tablet_bootstrap.cc:492] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: No bootstrap required, opened a new log
I20260927 21:07:36.910427 8405 ts_tablet_manager.cc:1397] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:07:36.910518 8405 raft_consensus.cc:348] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.910583 8405 raft_consensus.cc:374] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.910604 8405 raft_consensus.cc:729] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.910641 8405 consensus_queue.cc:260] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.910725 8405 ts_tablet_manager.cc:1428] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.911201 8407 tablet_bootstrap.cc:492] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Bootstrap starting.
I20260927 21:07:36.911485 8406 tablet_bootstrap.cc:492] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: No bootstrap required, opened a new log
I20260927 21:07:36.911490 8407 tablet_bootstrap.cc:654] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:36.911535 8406 ts_tablet_manager.cc:1397] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:07:36.911633 8406 raft_consensus.cc:348] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.911697 8406 raft_consensus.cc:374] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.911717 8406 raft_consensus.cc:729] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c524ed5fec4ae4931da965b63c9a9b, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.911752 8406 consensus_queue.cc:260] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.911832 8406 ts_tablet_manager.cc:1428] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.912209 8407 tablet_bootstrap.cc:492] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: No bootstrap required, opened a new log
I20260927 21:07:36.912258 8407 ts_tablet_manager.cc:1397] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:36.912371 8407 raft_consensus.cc:348] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.912438 8407 raft_consensus.cc:374] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:36.912456 8407 raft_consensus.cc:729] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3d6716f00f423e88dc3d608ed818a7, State: Initialized, Role: FOLLOWER
I20260927 21:07:36.912496 8407 consensus_queue.cc:260] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.912580 8407 ts_tablet_manager.cc:1428] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:36.930473 8411 raft_consensus.cc:482] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:36.930558 8411 raft_consensus.cc:504] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.930692 8411 leader_election.cc:288] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825), 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343)
I20260927 21:07:36.930835 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30705240040b438f832c987d23804405" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3d6716f00f423e88dc3d608ed818a7" is_pre_election: true
I20260927 21:07:36.930944 8191 raft_consensus.cc:2454] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 0.
I20260927 21:07:36.930925 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30705240040b438f832c987d23804405" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b" is_pre_election: true
I20260927 21:07:36.931006 8051 raft_consensus.cc:2454] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 0.
I20260927 21:07:36.931089 8262 leader_election.cc:302] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6290af3ade044bb18f19ee749ebe5c8a, 7c3d6716f00f423e88dc3d608ed818a7; no voters:
I20260927 21:07:36.931203 8411 raft_consensus.cc:2781] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:36.931255 8411 raft_consensus.cc:482] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:36.931277 8411 raft_consensus.cc:3037] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.931991 8411 raft_consensus.cc:504] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.932122 8411 leader_election.cc:288] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 election: Requested vote from peers 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825), 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343)
I20260927 21:07:36.932284 8051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30705240040b438f832c987d23804405" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c524ed5fec4ae4931da965b63c9a9b"
I20260927 21:07:36.932271 8191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30705240040b438f832c987d23804405" candidate_uuid: "6290af3ade044bb18f19ee749ebe5c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3d6716f00f423e88dc3d608ed818a7"
I20260927 21:07:36.932348 8051 raft_consensus.cc:3037] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.932348 8191 raft_consensus.cc:3037] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:36.933001 8051 raft_consensus.cc:2454] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 1.
I20260927 21:07:36.933024 8191 raft_consensus.cc:2454] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6290af3ade044bb18f19ee749ebe5c8a in term 1.
I20260927 21:07:36.933156 8265 leader_election.cc:302] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c524ed5fec4ae4931da965b63c9a9b, 6290af3ade044bb18f19ee749ebe5c8a; no voters:
I20260927 21:07:36.933235 8411 raft_consensus.cc:2781] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:36.933342 8411 raft_consensus.cc:686] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 LEADER]: Becoming Leader. State: Replica: 6290af3ade044bb18f19ee749ebe5c8a, State: Running, Role: LEADER
I20260927 21:07:36.933396 8411 consensus_queue.cc:237] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } }
I20260927 21:07:36.933907 7893 catalog_manager.cc:5495] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6290af3ade044bb18f19ee749ebe5c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6290af3ade044bb18f19ee749ebe5c8a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36431 } health_report { overall_health: HEALTHY } } }
I20260927 21:07:36.945179 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.942778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.0.98.3:36431)
0927 21:07:36.942815 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.942817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.942826 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:36.942874 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.942875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.942885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.942949 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.943028 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.943804 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.944498 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.944790 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.944861 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.944968 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.945000 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.945048 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.945091 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.945109 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:07:36.949798 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.947369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41659 (local address 127.0.98.2:43343)
0927 21:07:36.947405 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.947407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.947540 (+ 133us) server_negotiation.cc:407] Connection header received
0927 21:07:36.947596 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.947597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.947606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.947645 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.947747 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.948192 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.949229 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.949399 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.949446 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.949623 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.949646 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.949682 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.949733 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.949749 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:07:36.949851 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.947291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43343 (local address 127.0.98.3:41659)
0927 21:07:36.947471 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.947480 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.947540 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.947655 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.947655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.947657 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:36.947696 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.947697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.948257 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.948257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.949151 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.949153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.949466 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.949467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.949530 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.949544 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.949683 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.949684 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.949732 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.949774 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260927 21:07:36.951821 8051 raft_consensus.cc:1261] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Refusing update from remote peer 6290af3ade044bb18f19ee749ebe5c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.951833 8191 raft_consensus.cc:1261] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Refusing update from remote peer 6290af3ade044bb18f19ee749ebe5c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:36.952029 8489 consensus_queue.cc:1035] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c524ed5fec4ae4931da965b63c9a9b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.952112 8411 consensus_queue.cc:1035] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3d6716f00f423e88dc3d608ed818a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:36.959163 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.956702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51761 (local address 127.0.98.3:36431)
0927 21:07:36.956759 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:36.956760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:36.956767 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:36.956785 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:36.956785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:36.956795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:36.956838 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:36.956961 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.957409 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.958428 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.958712 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.958768 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.958920 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.958946 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.958987 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:36.959040 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:07:36.959054 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:07:36.959168 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.956556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36431 (local address 127.0.98.2:51761)
0927 21:07:36.956587 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.956597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:36.956635 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:36.956872 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:36.956873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:36.956874 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:07:36.956909 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.956910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.957485 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.957486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.958335 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:36.958337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:36.958796 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:36.958797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:36.958849 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:36.958855 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:07:36.958989 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:07:36.958990 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:07:36.959038 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:07:36.959099 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 21:07:36.964668 7893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.964756 7893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:36.965348 7893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d589e2c89ce4a739f77fac4d3a0936f: Sending DeleteTablet for 3 replicas of tablet 30705240040b438f832c987d23804405
I20260927 21:07:36.965556 8031 tablet_service.cc:1514] Processing DeleteTablet for tablet 30705240040b438f832c987d23804405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54052
I20260927 21:07:36.965598 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet 30705240040b438f832c987d23804405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53342
I20260927 21:07:36.965586 8311 tablet_service.cc:1514] Processing DeleteTablet for tablet 30705240040b438f832c987d23804405 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50968
I20260927 21:07:36.965682 8428 tablet_replica.cc:330] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: stopping tablet replica
I20260927 21:07:36.965688 8429 tablet_replica.cc:330] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: stopping tablet replica
I20260927 21:07:36.965734 8428 raft_consensus.cc:2229] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.965744 8429 raft_consensus.cc:2229] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:36.965763 8428 raft_consensus.cc:2258] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.965778 8429 raft_consensus.cc:2258] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.965826 8427 tablet_replica.cc:330] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: stopping tablet replica
I20260927 21:07:36.965875 8427 raft_consensus.cc:2229] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:36.965929 8427 raft_consensus.cc:2258] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:36.966068 8429 ts_tablet_manager.cc:1905] T 30705240040b438f832c987d23804405 P 60c524ed5fec4ae4931da965b63c9a9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.966673 8427 ts_tablet_manager.cc:1905] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.966876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 7966
I20260927 21:07:36.967906 8427 ts_tablet_manager.cc:1918] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:36.967950 8427 log.cc:1196] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/30705240040b438f832c987d23804405
I20260927 21:07:36.968148 8427 ts_tablet_manager.cc:1939] T 30705240040b438f832c987d23804405 P 6290af3ade044bb18f19ee749ebe5c8a: Deleting consensus metadata
I20260927 21:07:36.968454 7881 catalog_manager.cc:4841] TS 6290af3ade044bb18f19ee749ebe5c8a (127.0.98.3:36431): tablet 30705240040b438f832c987d23804405 (table test-table [id=d3fae82eedb34446b5034916b40a4d2b]) successfully deleted
I20260927 21:07:36.971629 8428 ts_tablet_manager.cc:1905] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:36.973278 8428 ts_tablet_manager.cc:1918] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:07:36.973341 8428 log.cc:1196] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/30705240040b438f832c987d23804405
I20260927 21:07:36.973559 8428 ts_tablet_manager.cc:1939] T 30705240040b438f832c987d23804405 P 7c3d6716f00f423e88dc3d608ed818a7: Deleting consensus metadata
I20260927 21:07:36.973878 7878 catalog_manager.cc:4841] TS 7c3d6716f00f423e88dc3d608ed818a7 (127.0.98.2:43343): tablet 30705240040b438f832c987d23804405 (table test-table [id=d3fae82eedb34446b5034916b40a4d2b]) successfully deleted
W20260927 21:07:36.973908 7881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34825 (error 108)
I20260927 21:07:36.974462 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:36.974274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.0.1:38448)
0927 21:07:36.974376 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:36.974417 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260927 21:07:36.974503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 8106
W20260927 21:07:36.974627 7881 catalog_manager.cc:4584] TS 60c524ed5fec4ae4931da965b63c9a9b (127.0.98.1:34825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30705240040b438f832c987d23804405: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34825: connect: Connection refused (error 111)
I20260927 21:07:36.980715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 8246
I20260927 21:07:36.987010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 7862
2026-09-27T21:07:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2828 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:37 dist-test-slave-8bgs krb5kdc[8501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:37 dist-test-slave-8bgs krb5kdc[8501](info): set up 2 sockets
Sep 27 21:07:37 dist-test-slave-8bgs krb5kdc[8501](info): commencing operation
krb5kdc: starting...
W20260927 21:07:38.008733 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:38 dist-test-slave-8bgs krb5kdc[8501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:38.117352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:38.193715 8514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:38.193988 8514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:38.194020 8514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:38.195353 8514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:38.195393 8514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:38.195407 8514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:38.195420 8514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:38.195454 8514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:38.196938 8514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:38.197166 8514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:38.197414 8514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:38.201994 8522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:38.202019 8520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:38.202036 8519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:38.202270 8514 server_base.cc:1034] running on GCE node
I20260927 21:07:38.202471 8514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:38.202723 8514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:38.203842 8514 hybrid_clock.cc:648] HybridClock initialized: now 1790543258203833 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T21:07:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1216 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:38 dist-test-slave-8bgs krb5kdc[8527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:38 dist-test-slave-8bgs krb5kdc[8527](info): set up 2 sockets
Sep 27 21:07:38 dist-test-slave-8bgs krb5kdc[8527](info): commencing operation
krb5kdc: starting...
W20260927 21:07:39.243755 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:39.355443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36817
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:39.433640 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:39.433846 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:39.433867 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:39.434964 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:39.435003 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:39.435016 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:39.435029 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:39.435047 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:39.436553 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:39.436741 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:39.436900 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:39.441435 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.441440 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.441457 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:39.441689 8540 server_base.cc:1034] running on GCE node
I20260927 21:07:39.441942 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:39.442260 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:39.443414 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790543259443406 us; error 33 us; skew 500 ppm
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:39.446148 8540 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:39.446614 8540 webserver.cc:460] Webserver started at http://127.0.98.62:40267/ using document root <none> and password file <none>
I20260927 21:07:39.446839 8540 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:39.446906 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:39.447048 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:39.448107 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "cddcf9bfbda04583a6c65832ba51c04d"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "86f7b22f0b53a54cb43ddbc8431e608a"
server_key_iv: "6e5047d21c388243f50c4320786697b8"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.448464 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "cddcf9bfbda04583a6c65832ba51c04d"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "86f7b22f0b53a54cb43ddbc8431e608a"
server_key_iv: "6e5047d21c388243f50c4320786697b8"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.449751 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:39.450431 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.450596 8540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:39.450690 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "cddcf9bfbda04583a6c65832ba51c04d"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "86f7b22f0b53a54cb43ddbc8431e608a"
server_key_iv: "6e5047d21c388243f50c4320786697b8"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.450773 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:39.467074 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:39.469412 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:39.469578 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:39.473495 8540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36817
I20260927 21:07:39.473529 8607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36817 every 8 connection(s)
I20260927 21:07:39.474009 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:39.474661 8608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:39.476869 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Bootstrap starting.
I20260927 21:07:39.477452 8608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:39.477744 8608 log.cc:824] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:39.478394 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: No bootstrap required, opened a new log
I20260927 21:07:39.479799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 8540
I20260927 21:07:39.479907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:39.480110 392 external_mini_cluster.cc:1428] Setting key acdd980521798f669e17f1e269344aa0
I20260927 21:07:39.480331 8608 raft_consensus.cc:348] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:39.480485 8608 raft_consensus.cc:374] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:39.480531 8608 raft_consensus.cc:729] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddcf9bfbda04583a6c65832ba51c04d, State: Initialized, Role: FOLLOWER
I20260927 21:07:39.480741 8608 consensus_queue.cc:260] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:39.480871 8608 raft_consensus.cc:388] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:39.480909 8608 raft_consensus.cc:482] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:39.480969 8608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:39.481594 8608 raft_consensus.cc:504] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:39.481716 8608 leader_election.cc:302] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cddcf9bfbda04583a6c65832ba51c04d; no voters:
I20260927 21:07:39.481920 8608 leader_election.cc:288] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:39.481946 8613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:39.482049 8613 raft_consensus.cc:686] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 LEADER]: Becoming Leader. State: Replica: cddcf9bfbda04583a6c65832ba51c04d, State: Running, Role: LEADER
I20260927 21:07:39.482183 8608 sys_catalog.cc:564] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:39.482183 8613 consensus_queue.cc:237] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:39.482682 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cddcf9bfbda04583a6c65832ba51c04d. Latest consensus state: current_term: 1 leader_uuid: "cddcf9bfbda04583a6c65832ba51c04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } } }
I20260927 21:07:39.482785 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:39.483240 8621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:39.483285 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cddcf9bfbda04583a6c65832ba51c04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } } }
I20260927 21:07:39.483459 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:39.483737 8621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:39.485337 8621 catalog_manager.cc:1347] Generated new cluster ID: f85506223b52412dad79994eef2a1514
I20260927 21:07:39.485389 8621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:39.491048 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.480691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.0.98.62:36817)
0927 21:07:39.481308 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:39.481316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:39.481340 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:07:39.481437 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:39.481442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:39.481490 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:39.481634 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:39.482407 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.483850 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.484867 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.485476 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.485571 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.487931 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:39.487936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:39.487936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:39.487947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:39.489558 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.489927 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.489930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.489930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.490030 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.490245 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.490246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.490247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.490476 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:39.490664 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:39.490730 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:07:39.490869 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":535,"threads_started":1}
I20260927 21:07:39.508002 8621 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:39.508555 8621 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:39.515218 8621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Generated new TSK 0
I20260927 21:07:39.515429 8621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:39.543303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36817
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:39.622138 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:39.622373 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:39.622414 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:39.623577 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:39.623641 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:39.623685 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:39.625129 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:39.625378 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:39.625603 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:39.630563 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.630541 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.630508 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:39.630735 8634 server_base.cc:1034] running on GCE node
I20260927 21:07:39.630981 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:39.631250 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:39.632416 8634 hybrid_clock.cc:648] HybridClock initialized: now 1790543259632399 us; error 43 us; skew 500 ppm
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:39.634855 8634 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:39.635438 8634 webserver.cc:460] Webserver started at http://127.0.98.1:38751/ using document root <none> and password file <none>
I20260927 21:07:39.635667 8634 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:39.635736 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:39.635875 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:39.636914 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "3b33172418314f2d95582e96d90a1287"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "c40c3775e92572ac63a5745a5dfcb152"
server_key_iv: "489b7f6f4976a623f3ed821685958288"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.637307 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3b33172418314f2d95582e96d90a1287"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "c40c3775e92572ac63a5745a5dfcb152"
server_key_iv: "489b7f6f4976a623f3ed821685958288"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.638679 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:39.639436 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.639604 8634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.639704 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "3b33172418314f2d95582e96d90a1287"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "c40c3775e92572ac63a5745a5dfcb152"
server_key_iv: "489b7f6f4976a623f3ed821685958288"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.639773 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:39.657207 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:39.659256 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:39.659412 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:39.659628 8634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:39.659940 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:39.659992 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.660027 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:39.660053 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.665380 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38387
I20260927 21:07:39.665454 8762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38387 every 8 connection(s)
I20260927 21:07:39.665756 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:39.667830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 8634
I20260927 21:07:39.667979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:39.668171 392 external_mini_cluster.cc:1428] Setting key ee261d5fc30f5886498f5e7077d69b78
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:07:39.676018 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.666513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50245 (local address 127.0.98.62:36817)
0927 21:07:39.666597 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:39.666599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:39.666732 (+ 133us) server_negotiation.cc:407] Connection header received
0927 21:07:39.667681 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:39.667683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:39.667702 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:39.667782 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:39.668584 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.669302 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.670326 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.670538 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.670630 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.673130 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:39.673131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:39.673132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:39.673148 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:39.674578 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.675021 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.675022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.675022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.675125 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.675376 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.675376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.675377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.675489 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:39.675565 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:39.675839 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:07:39.675924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:07:39.676223 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.666471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:50245)
0927 21:07:39.666616 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:39.666649 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:39.667555 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:39.667850 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:39.667854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:39.668183 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:39.668475 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:39.668482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.669393 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.669394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:39.670236 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:39.670238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.670639 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.670640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:39.670768 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.671315 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:39.671326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:39.673019 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:39.674688 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:39.674695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:39.674697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:39.674934 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:39.675178 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:39.675179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:39.675179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:39.675292 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:39.675557 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:39.675563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:39.675706 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 21:07:39.675817 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:07:39.675988 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260927 21:07:39.676741 8763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36817
I20260927 21:07:39.676879 8763 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:39.677048 8763 heartbeater.cc:511] Master 127.0.98.62:36817 requested a full tablet report, sending...
I20260927 21:07:39.677536 8572 ts_manager.cc:194] Registered new tserver with Master: 3b33172418314f2d95582e96d90a1287 (127.0.98.1:38387)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 21:07:39.678329 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:39.692646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36817
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:39.771332 8768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:39.771535 8768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:39.771559 8768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:39.772651 8768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:39.772686 8768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:39.772710 8768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:39.774092 8768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:39.774260 8768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:39.774428 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:39.779343 8768 server_base.cc:1034] running on GCE node
W20260927 21:07:39.779326 8776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.779307 8773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.779302 8774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:39.779850 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:39.780099 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:39.781263 8768 hybrid_clock.cc:648] HybridClock initialized: now 1790543259781249 us; error 25 us; skew 500 ppm
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:39.783911 8768 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:39.784302 8768 webserver.cc:460] Webserver started at http://127.0.98.2:43853/ using document root <none> and password file <none>
I20260927 21:07:39.784510 8768 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:39.784576 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:39.784721 8768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:39.785741 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "6236e48f3bd04b899af411adb38b8130"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "ec7db45fca3116b9728843248b8d837d"
server_key_iv: "e0493fa19be6c29bba2946b06bd1f539"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.786089 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6236e48f3bd04b899af411adb38b8130"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "ec7db45fca3116b9728843248b8d837d"
server_key_iv: "e0493fa19be6c29bba2946b06bd1f539"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.787477 8768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:07:39.788184 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.788341 8768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:39.788435 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "6236e48f3bd04b899af411adb38b8130"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "ec7db45fca3116b9728843248b8d837d"
server_key_iv: "e0493fa19be6c29bba2946b06bd1f539"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.788514 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:39.805459 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:39.807832 8768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:39.807994 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:39.808235 8768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:39.808534 8768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:39.808593 8768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.808627 8768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:39.808653 8768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.814144 8768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45915
I20260927 21:07:39.814215 8896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45915 every 8 connection(s)
I20260927 21:07:39.814535 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:39.816819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 8768
I20260927 21:07:39.816907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:39.817114 392 external_mini_cluster.cc:1428] Setting key c6579e75e01b3c9358a2690ea1a7a957
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:39.824461 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.815291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45195 (local address 127.0.98.62:36817)
0927 21:07:39.815364 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:39.815366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:39.815584 (+ 218us) server_negotiation.cc:407] Connection header received
0927 21:07:39.816548 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:39.816549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:39.816562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:39.816625 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:39.817391 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.817904 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.819013 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.819272 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.819331 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.821664 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:39.821666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:39.821666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:39.821674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:39.823046 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.823515 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.823516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.823517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.823575 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.823848 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.823849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.823849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.823961 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:39.824033 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:39.824294 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:07:39.824374 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:07:39.824684 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.815287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:45195)
0927 21:07:39.815451 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:39.815487 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:39.816428 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:39.816712 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:39.816716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:39.817042 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:39.817294 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:39.817299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.818001 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.818002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:39.818924 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:39.818928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.819352 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.819353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:39.819507 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.819984 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:39.819993 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:39.821570 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:39.823192 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:39.823198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:39.823200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:39.823437 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:39.823639 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:39.823640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:39.823641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:39.823754 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:39.824024 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:39.824029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:39.824168 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:07:39.824274 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:07:39.824423 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260927 21:07:39.825098 8897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36817
I20260927 21:07:39.825215 8897 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:39.825389 8897 heartbeater.cc:511] Master 127.0.98.62:36817 requested a full tablet report, sending...
I20260927 21:07:39.825773 8572 ts_manager.cc:194] Registered new tserver with Master: 6236e48f3bd04b899af411adb38b8130 (127.0.98.2:45915)
I20260927 21:07:39.826143 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:39.841822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36817
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:39.920291 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:39.920478 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:39.920501 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:39.921657 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:39.921698 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:39.921725 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:39.923230 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:39.923480 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:39.923705 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:39.928610 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.928639 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:39.928689 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:39.928732 8902 server_base.cc:1034] running on GCE node
I20260927 21:07:39.929047 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:39.929279 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:39.930416 8902 hybrid_clock.cc:648] HybridClock initialized: now 1790543259930398 us; error 36 us; skew 500 ppm
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:39.932869 8902 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:39.933274 8902 webserver.cc:460] Webserver started at http://127.0.98.3:35895/ using document root <none> and password file <none>
I20260927 21:07:39.933482 8902 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:39.933525 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:39.933684 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:39.934681 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "30843e50ef67442f9d60ca1cd7fd7fe5"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "bb374d107d7090557781e49daa35d79c"
server_key_iv: "54130f99bf1e9ff8994ff71ed7adaf64"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.935027 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30843e50ef67442f9d60ca1cd7fd7fe5"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "bb374d107d7090557781e49daa35d79c"
server_key_iv: "54130f99bf1e9ff8994ff71ed7adaf64"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.936322 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:39.936980 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.937155 8902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:39.937238 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "30843e50ef67442f9d60ca1cd7fd7fe5"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "bb374d107d7090557781e49daa35d79c"
server_key_iv: "54130f99bf1e9ff8994ff71ed7adaf64"
server_key_version: "encryptionkey@0"
I20260927 21:07:39.937290 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:39.949548 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:39.951424 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:39.951537 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:39.951694 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:39.951993 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:39.952050 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.952097 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:39.952127 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:39.957650 8902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35615
I20260927 21:07:39.957687 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35615 every 8 connection(s)
I20260927 21:07:39.958016 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:39 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:39.966421 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.958730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51233 (local address 127.0.98.62:36817)
0927 21:07:39.958806 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:39.958808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:39.958981 (+ 173us) server_negotiation.cc:407] Connection header received
0927 21:07:39.959943 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:39.959944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:39.959953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:39.960004 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:39.960554 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.961034 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.961966 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.962142 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.962190 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.964179 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:39.964180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:39.964180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:39.964187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:39.965324 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.965678 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.965679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.965679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.965726 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:39.965928 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:39.965929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:39.965929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:39.966023 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:39.966071 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:39.966304 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:07:39.966359 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:07:39.966645 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:39.958691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:51233)
0927 21:07:39.958866 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:39.958901 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:39.959851 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:39.960056 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:39.960060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:39.960301 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:39.960479 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:39.960484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.961110 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.961111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:39.961893 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:39.961896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:39.962211 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:39.962211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:39.962303 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:39.962723 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:39.962731 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:39.964093 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:39.965395 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:39.965400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:39.965402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:39.965614 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:39.965782 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:39.965783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:39.965783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:39.965875 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:39.966085 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:39.966088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:39.966196 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 21:07:39.966277 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:07:39.966391 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260927 21:07:39.966697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 8902
I20260927 21:07:39.966821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:39.967034 392 external_mini_cluster.cc:1428] Setting key 911d673a575aba7f5dabceb7801ffdb6
I20260927 21:07:39.967204 9031 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36817
I20260927 21:07:39.967303 9031 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:39.967515 9031 heartbeater.cc:511] Master 127.0.98.62:36817 requested a full tablet report, sending...
I20260927 21:07:39.967893 8572 ts_manager.cc:194] Registered new tserver with Master: 30843e50ef67442f9d60ca1cd7fd7fe5 (127.0.98.3:35615)
I20260927 21:07:39.968225 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51233
I20260927 21:07:39.978878 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 21:07:40.045070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 8634
I20260927 21:07:40.049512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 8768
I20260927 21:07:40.053862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 8902
I20260927 21:07:40.057837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 8540
I20260927 21:07:40.062628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36817
--webserver_interface=127.0.98.62
--webserver_port=40267
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:40.140727 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:40.140911 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:40.140931 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:40.142096 9040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:40.142133 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:40.142148 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:40.142159 9040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:40.142211 9040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:40.143661 9040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:40.143899 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:40.144116 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:07:40.148768 9040 server_base.cc:1034] running on GCE node
W20260927 21:07:40.148782 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:40.148759 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:40.148777 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:40.149315 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:40.149592 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:40.150751 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790543260150741 us; error 29 us; skew 500 ppm
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:40.153048 9040 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:40.153437 9040 webserver.cc:460] Webserver started at http://127.0.98.62:40267/ using document root <none> and password file <none>
I20260927 21:07:40.153659 9040 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:40.153738 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:40.154991 9040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:40.155643 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.155800 9040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.155900 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "cddcf9bfbda04583a6c65832ba51c04d"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "86f7b22f0b53a54cb43ddbc8431e608a"
server_key_iv: "6e5047d21c388243f50c4320786697b8"
server_key_version: "encryptionkey@0"
I20260927 21:07:40.156216 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:40.176754 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:40.178491 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:40.178655 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:40.182504 9040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36817
I20260927 21:07:40.182538 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36817 every 8 connection(s)
I20260927 21:07:40.183053 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:40.184242 9108 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:07:40.184932 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Bootstrap starting.
I20260927 21:07:40.187655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9040
I20260927 21:07:40.187988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38387
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36817
--webserver_port=38751
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:07:40.194008 9108 log.cc:824] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:40.196269 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:07:40.196594 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Bootstrap complete.
I20260927 21:07:40.200475 9108 raft_consensus.cc:348] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:40.200764 9108 raft_consensus.cc:729] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddcf9bfbda04583a6c65832ba51c04d, State: Initialized, Role: FOLLOWER
I20260927 21:07:40.200954 9108 consensus_queue.cc:260] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:40.201020 9108 raft_consensus.cc:388] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:40.201058 9108 raft_consensus.cc:482] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:40.201113 9108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:07:40.202378 9108 raft_consensus.cc:504] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:40.202488 9108 leader_election.cc:302] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cddcf9bfbda04583a6c65832ba51c04d; no voters:
I20260927 21:07:40.202641 9108 leader_election.cc:288] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:07:40.202709 9112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:07:40.202893 9112 raft_consensus.cc:686] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [term 2 LEADER]: Becoming Leader. State: Replica: cddcf9bfbda04583a6c65832ba51c04d, State: Running, Role: LEADER
I20260927 21:07:40.202879 9108 sys_catalog.cc:564] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:40.203697 9112 consensus_queue.cc:237] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } }
I20260927 21:07:40.204249 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cddcf9bfbda04583a6c65832ba51c04d. Latest consensus state: current_term: 2 leader_uuid: "cddcf9bfbda04583a6c65832ba51c04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } } }
I20260927 21:07:40.204247 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cddcf9bfbda04583a6c65832ba51c04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddcf9bfbda04583a6c65832ba51c04d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36817 } } }
I20260927 21:07:40.204356 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:40.204701 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:40.205087 9126 catalog_manager.cc:1259] Loaded cluster ID: f85506223b52412dad79994eef2a1514
I20260927 21:07:40.205132 9126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: loading cluster ID for follower catalog manager: success
I20260927 21:07:40.206245 9126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: acquiring CA information for follower catalog manager: success
I20260927 21:07:40.206563 9126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:07:40.206677 9128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:40.206909 9128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:40.206991 9128 catalog_manager.cc:1259] Loaded cluster ID: f85506223b52412dad79994eef2a1514
I20260927 21:07:40.207015 9128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:40.207244 9128 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:40.207322 9128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cddcf9bfbda04583a6c65832ba51c04d: Loaded TSK: 0
I20260927 21:07:40.207444 9128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:07:40.273926 9110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:40.274119 9110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:40.274142 9110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:40.275364 9110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:40.275405 9110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:40.275432 9110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:40.276857 9110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38751
--tserver_master_addrs=127.0.98.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:40.277087 9110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:40.277448 9110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:40.282318 9135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:40.282311 9137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:40.282312 9134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:40.282562 9110 server_base.cc:1034] running on GCE node
I20260927 21:07:40.282904 9110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:40.283195 9110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:40.284344 9110 hybrid_clock.cc:648] HybridClock initialized: now 1790543260284329 us; error 31 us; skew 500 ppm
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:40.286791 9110 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 21:07:40.287324 9110 webserver.cc:460] Webserver started at http://127.0.98.1:38751/ using document root <none> and password file <none>
I20260927 21:07:40.287575 9110 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:40.287631 9110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:40.288921 9110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:40.289527 9144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.289676 9110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:40.289757 9110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "3b33172418314f2d95582e96d90a1287"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "c40c3775e92572ac63a5745a5dfcb152"
server_key_iv: "489b7f6f4976a623f3ed821685958288"
server_key_version: "encryptionkey@0"
I20260927 21:07:40.290084 9110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:40.309182 9110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:40.311352 9110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:40.311494 9110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:40.311759 9110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:40.312134 9110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:40.312170 9110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.312192 9110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:40.312206 9110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.319192 9110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38387
I20260927 21:07:40.319229 9257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38387 every 8 connection(s)
I20260927 21:07:40.319566 9110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:40.324553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9110
I20260927 21:07:40.324842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45915
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36817
--webserver_port=43853
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:40.329569 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.320260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:54785)
0927 21:07:40.320447 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.320494 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.320951 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.321388 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.321391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.321929 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.322114 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.322119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.323686 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.323687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.324497 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.324500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.326379 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.326380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.326525 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.327123 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.327135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.329357 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
W20260927 21:07:40.329615 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.320317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54785 (local address 127.0.98.62:36817)
0927 21:07:40.320582 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.320594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.320645 (+ 51us) server_negotiation.cc:407] Connection header received
0927 21:07:40.321146 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.321154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.321210 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.321351 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.322235 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.323600 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.324598 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.326183 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.326289 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.329473 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54785: BlockingRecv error: recv got EOF from 127.0.98.1:54785 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
W20260927 21:07:40.329871 9258 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36817 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36817: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:40.334185 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.330096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:48855)
0927 21:07:40.330133 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.330145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.330248 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.330379 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.330380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.330506 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.330549 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.330551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.331221 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.331222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.331719 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.331721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.332087 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.332088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.332153 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.332568 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.332573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.334107 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 21:07:40.334244 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.330177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48855 (local address 127.0.98.62:36817)
0927 21:07:40.330208 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.330210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.330232 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:07:40.330309 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.330310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.330321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.330376 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.330629 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.331080 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.331811 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.332004 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.332087 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.334184 (+ 2097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48855: BlockingRecv error: recv got EOF from 127.0.98.1:48855 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:07:40.338477 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.334578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50223 (local address 127.0.98.62:36817)
0927 21:07:40.334634 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.334637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.334646 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:40.334665 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.334666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.334675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.334723 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.334938 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.335448 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.336102 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.336292 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.336363 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.338440 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50223: BlockingRecv error: recv got EOF from 127.0.98.1:50223 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:07:40.338474 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.334504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:50223)
0927 21:07:40.334547 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.334557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.334610 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.334724 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.334725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.334845 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.334868 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.334869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.335521 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.335523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.336006 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.336007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.336361 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.336362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.336412 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.336810 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.336814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.338370 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 21:07:40.338616 9258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:07:40.406822 9262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:40.407016 9262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:40.407038 9262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:40.408262 9262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:40.408331 9262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:40.408370 9262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:40.409842 9262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43853
--tserver_master_addrs=127.0.98.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:40.410015 9262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:40.410178 9262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:40.414898 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:40.414918 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:40.415050 9262 server_base.cc:1034] running on GCE node
W20260927 21:07:40.414947 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:40.415347 9262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:40.415546 9262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:40.416703 9262 hybrid_clock.cc:648] HybridClock initialized: now 1790543260416700 us; error 38 us; skew 500 ppm
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:40.419026 9262 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 21:07:40.419499 9262 webserver.cc:460] Webserver started at http://127.0.98.2:43853/ using document root <none> and password file <none>
I20260927 21:07:40.419790 9262 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:40.419875 9262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:40.421216 9262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:40.421773 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.421939 9262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.422032 9262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "6236e48f3bd04b899af411adb38b8130"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "ec7db45fca3116b9728843248b8d837d"
server_key_iv: "e0493fa19be6c29bba2946b06bd1f539"
server_key_version: "encryptionkey@0"
I20260927 21:07:40.422380 9262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:40.438199 9262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:40.440058 9262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:40.440210 9262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:40.440426 9262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:40.440723 9262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:40.440775 9262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.440815 9262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:40.440842 9262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.446384 9262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45915
I20260927 21:07:40.446424 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45915 every 8 connection(s)
I20260927 21:07:40.446779 9262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:40.450208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9262
I20260927 21:07:40.450490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35615
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36817
--webserver_port=35895
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:07:40.454200 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.447507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54115 (local address 127.0.98.62:36817)
0927 21:07:40.447592 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.447594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.447773 (+ 179us) server_negotiation.cc:407] Connection header received
0927 21:07:40.448220 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.448221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.448233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.448283 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.449127 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.449600 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.450915 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.451179 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.451274 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.454143 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54115: BlockingRecv error: recv got EOF from 127.0.98.2:54115 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:07:40.454258 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.447475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:54115)
0927 21:07:40.447656 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.447690 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.448140 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.448334 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.448338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.448872 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.449052 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.449056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.449673 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.449674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.450823 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.450825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.451294 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.451295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.451454 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.451931 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.451940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.454073 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
W20260927 21:07:40.454445 9391 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36817 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36817: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:40.458958 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.454692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:58859)
0927 21:07:40.454732 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.454741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.454795 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.454910 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.454911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.455043 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.455081 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.455119 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.455710 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.455711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.456213 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.456215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.456583 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.456584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.456646 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.457058 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.457063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.458880 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:07:40.459003 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.454761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58859 (local address 127.0.98.62:36817)
0927 21:07:40.454821 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.454823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.454831 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:40.454852 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.454853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.454861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.454910 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.455178 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.455633 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.456303 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.456500 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.456582 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.458942 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58859: BlockingRecv error: recv got EOF from 127.0.98.2:58859 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:40.463447 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.459319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:58151)
0927 21:07:40.459364 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.459374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.459408 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.459528 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.459529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.459657 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.459681 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.459682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.460280 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.460282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.460772 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.460774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.461168 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.461169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.461240 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.461648 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.461653 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.463373 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:07:40.463490 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.459400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58151 (local address 127.0.98.62:36817)
0927 21:07:40.459445 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.459447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.459454 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:40.459472 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.459473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.459481 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.459528 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.459755 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.460206 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.460851 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.461092 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.461168 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.463433 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58151: BlockingRecv error: recv got EOF from 127.0.98.2:58151 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:07:40.463701 9391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:07:40.533274 9394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:40.533476 9394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:40.533499 9394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:40.534641 9394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:40.534682 9394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:40.534708 9394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:40.536188 9394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35895
--tserver_master_addrs=127.0.98.62:36817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:40.536427 9394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:40.536650 9394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:40.541411 9399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:40.541527 9394 server_base.cc:1034] running on GCE node
W20260927 21:07:40.541422 9400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:40.541422 9402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:40.541911 9394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:40.542148 9394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:40.543352 9394 hybrid_clock.cc:648] HybridClock initialized: now 1790543260543340 us; error 43 us; skew 500 ppm
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:40.545701 9394 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 21:07:40.546093 9394 webserver.cc:460] Webserver started at http://127.0.98.3:35895/ using document root <none> and password file <none>
I20260927 21:07:40.546304 9394 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:40.546383 9394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:40.547744 9394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:40.548297 9409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.548445 9394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:40.548549 9394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "30843e50ef67442f9d60ca1cd7fd7fe5"
format_stamp: "Formatted at 2026-09-27 21:07:39 on dist-test-slave-8bgs"
server_key: "bb374d107d7090557781e49daa35d79c"
server_key_iv: "54130f99bf1e9ff8994ff71ed7adaf64"
server_key_version: "encryptionkey@0"
I20260927 21:07:40.548870 9394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:40.563797 9394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:40.566020 9394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:40.566179 9394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:40.566403 9394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:40.566697 9394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:40.566748 9394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.566782 9394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:40.566809 9394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:40.572317 9394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35615
I20260927 21:07:40.572391 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35615 every 8 connection(s)
I20260927 21:07:40.572724 9394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:40.575515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9394
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:07:40.580366 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.573429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41437 (local address 127.0.98.62:36817)
0927 21:07:40.573489 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.573492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.573731 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:07:40.574181 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.574182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.574193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.574244 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.575129 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.575639 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.577030 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.577317 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.577385 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.580299 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41437: BlockingRecv error: recv got EOF from 127.0.98.3:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:07:40.580374 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.573430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:41437)
0927 21:07:40.573617 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.573650 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.574105 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.574293 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.574296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.574824 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.575007 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.575011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.575733 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.575734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.576940 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.576942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.577395 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.577396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.577490 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.577968 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.577975 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.580205 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
W20260927 21:07:40.580678 9523 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36817 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36817: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 21:07:40.583549 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.576084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.62:36817)
0927 21:07:40.576173 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.576175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.576185 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:40.576260 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.576261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.576269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.576322 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.577030 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.577630 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.578391 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.578594 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.578670 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.580064 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:40.580068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:40.580068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:40.580079 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:40.582045 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:40.582428 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:40.582431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:40.582432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:40.582533 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:40.582752 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:40.582753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:40.582754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:40.583043 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:40.583204 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:40.583369 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:07:40.583454 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":46,"threads_started":1}
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:40.585013 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.580892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:58939)
0927 21:07:40.580938 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.580948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.581007 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.581138 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.581139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.581260 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.581301 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.581302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.581902 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.581903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.582388 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.582389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.582737 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.582738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.582806 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.583221 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.583226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.584939 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:07:40.585075 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.580950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58939 (local address 127.0.98.62:36817)
0927 21:07:40.581008 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.581010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.581018 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:40.581053 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.581054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.581063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.581134 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.581373 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.581825 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.582468 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.582656 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.582736 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.585002 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58939: BlockingRecv error: recv got EOF from 127.0.98.3:58939 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:40 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:40.588963 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:40.585408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:41821)
0927 21:07:40.585457 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:40.585467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:40.585507 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:40.585610 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:40.585611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:40.585711 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:40.585735 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.585736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.586261 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.586261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.586725 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:40.586726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.587001 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.587001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:40.587035 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.587456 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:40.587461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:40.588890 (+ 1429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:07:40.589004 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:40.585473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41821 (local address 127.0.98.62:36817)
0927 21:07:40.585523 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:40.585525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:40.585533 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:40.585551 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:40.585552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:40.585560 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:40.585608 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:40.585787 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.586203 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.586787 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:40.586945 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:40.586989 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:40.588950 (+ 1961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41821: BlockingRecv error: recv got EOF from 127.0.98.3:41821 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 21:07:40.589134 9523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 21:07:41 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:41 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:41.343199 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:41.338966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:52345)
0927 21:07:41.339155 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:41.339165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:41.339214 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:41.339353 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:41.339354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:41.339501 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:41.339547 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:41.339549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.340161 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.340162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:41.340676 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:41.340677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.340980 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.340980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:41.341025 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:41.341462 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:41.341466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:41.343131 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
W20260927 21:07:41.343310 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:41.339045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52345 (local address 127.0.98.62:36817)
0927 21:07:41.339193 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:41.339194 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:41.339202 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:41.339270 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:41.339270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:41.339280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:41.339337 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:41.339610 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.340102 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.340744 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.340925 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.340973 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:41.343208 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52345: BlockingRecv error: recv got EOF from 127.0.98.1:52345 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
Sep 27 21:07:41 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:41 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:41.471306 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:41.464067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:57161)
0927 21:07:41.464193 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:41.464211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:41.464263 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:41.464380 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:41.464381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:41.464531 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:41.464578 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:41.464579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.465172 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.465172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:41.465695 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:41.465696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.465993 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.465994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:41.466039 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:41.469459 (+ 3420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:41.469463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:41.471239 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
W20260927 21:07:41.471431 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:41.464112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57161 (local address 127.0.98.62:36817)
0927 21:07:41.464183 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:41.464185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:41.464262 (+ 77us) server_negotiation.cc:407] Connection header received
0927 21:07:41.464303 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:41.464303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:41.464313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:41.464363 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:41.464632 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.465110 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.465758 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.465934 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.465983 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:41.471331 (+ 5348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57161: BlockingRecv error: recv got EOF from 127.0.98.2:57161 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 21:07:41 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:41 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:41.594174 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:41.589586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:58817)
0927 21:07:41.589752 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:41.589763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:41.589844 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:41.589993 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:41.589993 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:41.590179 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:41.590222 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:41.590224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.590880 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.590881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:41.591439 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:41.591440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.591801 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.591802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:41.591863 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:41.592333 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:41.592337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:41.594068 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
W20260927 21:07:41.594203 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:41.589671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58817 (local address 127.0.98.62:36817)
0927 21:07:41.589744 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:41.589746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:41.589829 (+ 83us) server_negotiation.cc:407] Connection header received
0927 21:07:41.589926 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:41.589927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:41.589937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:41.589978 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:41.590296 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.590799 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.591522 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:41.591725 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:41.591801 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:41.594138 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58817: BlockingRecv error: recv got EOF from 127.0.98.3:58817 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 21:07:42 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:42 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:42.348063 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:42.343601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:49645)
0927 21:07:42.343765 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:42.343776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:42.343834 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:42.343966 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:42.343967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:42.344122 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:42.344167 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:42.344169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.344795 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.344796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:42.345303 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:42.345305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.345624 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.345625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:42.345663 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:42.346090 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:42.346095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:42.348004 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
W20260927 21:07:42.348163 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:42.343673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49645 (local address 127.0.98.62:36817)
0927 21:07:42.343778 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:42.343781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:42.343821 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:07:42.343887 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:42.343888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:42.343897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:42.343954 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:42.344234 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.344735 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.345376 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.345565 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.345614 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:42.348092 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49645: BlockingRecv error: recv got EOF from 127.0.98.1:49645 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":45,"threads_started":1}
Sep 27 21:07:42 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:42 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:42.476130 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:42.471872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:34579)
0927 21:07:42.471979 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:42.471996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:42.472051 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:42.472181 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:42.472181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:42.472334 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:42.472381 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:42.472382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.472977 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.472977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:42.473477 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:42.473478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.473781 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.473782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:42.473819 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:42.474230 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:42.474234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:42.476074 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
W20260927 21:07:42.476279 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:42.471934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34579 (local address 127.0.98.62:36817)
0927 21:07:42.472004 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:42.472006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:42.472037 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:07:42.472094 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:42.472095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:42.472105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:42.472167 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:42.472436 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.472913 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.473544 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.473721 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.473770 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:42.476195 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34579: BlockingRecv error: recv got EOF from 127.0.98.2:34579 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 21:07:42 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:42 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:42.599164 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:42.594626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:47405)
0927 21:07:42.594759 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:42.594770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:42.594814 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:42.594941 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:42.594942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:42.595183 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:42.595232 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:42.595234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.595883 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.595884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:42.596410 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:42.596411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.596742 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.596743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:42.596783 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:42.597216 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:42.597220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:42.599045 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
W20260927 21:07:42.599211 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:42.594758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47405 (local address 127.0.98.62:36817)
0927 21:07:42.594808 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:42.594809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:42.594817 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:42.594876 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:42.594877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:42.594887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:42.594937 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:42.595300 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.595780 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.596488 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:42.596678 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:42.596732 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:42.599145 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47405: BlockingRecv error: recv got EOF from 127.0.98.3:47405 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 21:07:43 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:43 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:43.352981 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:43.348500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:48255)
0927 21:07:43.348630 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:43.348642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:43.348701 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:43.348818 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:43.348819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:43.348970 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:43.349020 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:43.349022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.349653 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.349654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:43.350171 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:43.350172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.350483 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.350484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:43.350523 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:43.350960 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:43.350965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:43.352915 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
W20260927 21:07:43.353080 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:43.348557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48255 (local address 127.0.98.62:36817)
0927 21:07:43.348674 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:43.348675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:43.348685 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:43.348745 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:43.348746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:43.348756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:43.348814 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:43.349079 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.349577 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.350240 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.350421 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.350469 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:43.353006 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48255: BlockingRecv error: recv got EOF from 127.0.98.1:48255 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
Sep 27 21:07:43 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:43 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:43.481051 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:43.476550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:42045)
0927 21:07:43.476690 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:43.476702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:43.476780 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:43.476923 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:43.476924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:43.477107 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:43.477154 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:43.477156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.477824 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.477825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:43.478351 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:43.478352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.478706 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.478707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:43.478764 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:43.479248 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:43.479252 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:43.480950 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260927 21:07:43.481099 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:43.476722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42045 (local address 127.0.98.62:36817)
0927 21:07:43.476791 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:43.476793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:43.476803 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:43.476833 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:43.476834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:43.476846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:43.476918 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:43.477235 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.477729 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.478434 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.478630 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.478709 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:43.481026 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42045: BlockingRecv error: recv got EOF from 127.0.98.2:42045 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 21:07:43 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:43 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:43.604733 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:43.599717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:44929)
0927 21:07:43.599887 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:43.599898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:43.599995 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:43.600179 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:43.600180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:43.600354 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:43.600400 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:43.600402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.601094 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.601095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:43.601689 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:43.601691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.602121 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.602123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:43.602200 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:43.602835 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:43.602840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:43.604630 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
W20260927 21:07:43.604763 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:43.599805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44929 (local address 127.0.98.62:36817)
0927 21:07:43.599881 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:43.599883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:43.599975 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:07:43.600088 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:43.600089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:43.600102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:43.600173 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:43.600479 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.601008 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.601780 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:43.602015 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:43.602084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:43.604700 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44929: BlockingRecv error: recv got EOF from 127.0.98.3:44929 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 21:07:44 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:44 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:44.357604 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:44.353442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:47471)
0927 21:07:44.353581 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:44.353600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:44.353655 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:44.353797 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:44.353797 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:44.353946 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:44.353993 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:44.353995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.354603 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.354603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:44.355132 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:44.355134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.355431 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.355432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:44.355470 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:44.355885 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:44.355889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:44.357541 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
W20260927 21:07:44.357730 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:44.353485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47471 (local address 127.0.98.62:36817)
0927 21:07:44.353613 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:44.353615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:44.353652 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:07:44.353703 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:44.353703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:44.353713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:44.353780 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:44.354050 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.354538 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.355193 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.355370 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.355420 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:44.357622 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47471: BlockingRecv error: recv got EOF from 127.0.98.1:47471 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
Sep 27 21:07:44 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:44 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:44.486308 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:44.481724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:39661)
0927 21:07:44.481864 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:44.481876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:44.481968 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:44.482131 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:44.482132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:44.482338 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:44.482386 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:44.482387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.483028 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.483029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:44.483639 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:44.483641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.484010 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.484011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:44.484067 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:44.484530 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:44.484535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:44.486213 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260927 21:07:44.486347 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:44.481752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39661 (local address 127.0.98.62:36817)
0927 21:07:44.481818 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:44.481820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:44.481986 (+ 166us) server_negotiation.cc:407] Connection header received
0927 21:07:44.482034 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:44.482035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:44.482057 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:44.482110 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:44.482455 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.482939 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.483730 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.483935 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.484012 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:44.486288 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39661: BlockingRecv error: recv got EOF from 127.0.98.2:39661 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 21:07:44 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:44 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:44.609848 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:44.605199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:57183)
0927 21:07:44.605365 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:44.605377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:44.605463 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:44.605610 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:44.605611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:44.605797 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:44.605854 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:44.605855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.606565 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.606567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:44.607145 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:44.607147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.607511 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.607512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:44.607572 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:44.608000 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:44.608005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:44.609754 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
W20260927 21:07:44.609910 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:44.605275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57183 (local address 127.0.98.62:36817)
0927 21:07:44.605348 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:44.605350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:44.605442 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:07:44.605516 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:44.605517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:44.605531 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:44.605604 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:44.605944 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.606473 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.607228 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:44.607424 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:44.607510 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:44.609826 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57183: BlockingRecv error: recv got EOF from 127.0.98.3:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 21:07:45 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:45 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:45.362593 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:45.358207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:56687)
0927 21:07:45.358373 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:45.358386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:45.358438 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:45.358576 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:45.358577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:45.358729 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:45.358776 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:45.358777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.359507 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.359508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:45.360074 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:45.360076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.360391 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.360392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:45.360430 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:45.360843 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:45.360847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:45.362536 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
W20260927 21:07:45.362723 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:45.358324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56687 (local address 127.0.98.62:36817)
0927 21:07:45.358433 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:45.358435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:45.358448 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:07:45.358485 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:45.358485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:45.358495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:45.358555 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:45.358838 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.359413 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.360143 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.360327 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.360376 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:45.362618 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56687: BlockingRecv error: recv got EOF from 127.0.98.1:56687 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
Sep 27 21:07:45 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:45 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:45.491128 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:45.486776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:36745)
0927 21:07:45.486942 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:45.486952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:45.487020 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:45.487190 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:45.487190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:45.487344 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:45.487391 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:45.487393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.487994 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.487994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:45.488513 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:45.488514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.488816 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.488816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:45.488853 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:45.489273 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:45.489277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:45.491013 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
W20260927 21:07:45.491247 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:45.486904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36745 (local address 127.0.98.62:36817)
0927 21:07:45.486949 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:45.486952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:45.487028 (+ 76us) server_negotiation.cc:407] Connection header received
0927 21:07:45.487073 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:45.487073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:45.487118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:45.487179 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:45.487455 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.487932 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.488582 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.488756 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.488804 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:45.491158 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36745: BlockingRecv error: recv got EOF from 127.0.98.2:36745 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 21:07:45 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:45 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:45.614928 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:45.610377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:42929)
0927 21:07:45.610547 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:45.610559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:45.610658 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:45.610810 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:45.610810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:45.610984 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:45.611039 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:45.611041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.611764 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.611765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:45.612328 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:45.612330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.612697 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.612698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:45.612759 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:45.613175 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:45.613179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:45.614835 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
W20260927 21:07:45.614975 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:45.610576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42929 (local address 127.0.98.62:36817)
0927 21:07:45.610663 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:45.610664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:45.610674 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:45.610719 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:45.610720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:45.610739 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:45.610804 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:45.611191 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.611683 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.612424 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:45.612614 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:45.612697 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:45.614910 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42929: BlockingRecv error: recv got EOF from 127.0.98.3:42929 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 21:07:46 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:46 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:46.367277 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:46.363016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:33039)
0927 21:07:46.363164 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:46.363176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:46.363232 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:46.363364 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:46.363364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:46.363520 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:46.363568 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:46.363570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.364192 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.364192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:46.364703 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:46.364705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.365012 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.365012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:46.365050 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:46.365479 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:46.365483 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:46.367212 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
W20260927 21:07:46.367401 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:46.363117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33039 (local address 127.0.98.62:36817)
0927 21:07:46.363231 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:46.363232 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:46.363242 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:46.363274 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:46.363274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:46.363283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:46.363349 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:46.363636 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.364129 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.364774 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.364952 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.365000 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:46.367297 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33039: BlockingRecv error: recv got EOF from 127.0.98.1:33039 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
Sep 27 21:07:46 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:46 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:46.500864 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:46.491598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:44417)
0927 21:07:46.491718 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:46.491729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:46.491788 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:46.491914 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:46.491914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:46.492069 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:46.492114 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:46.492115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.492823 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.492823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:46.493343 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:46.493345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.493697 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.493698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:46.493783 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:46.498956 (+ 5173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:46.498962 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:46.500795 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
W20260927 21:07:46.500989 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:46.491668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44417 (local address 127.0.98.62:36817)
0927 21:07:46.491746 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:46.491748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:46.491771 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:07:46.491845 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:46.491846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:46.491856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:46.491903 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:46.492169 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.492758 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.493425 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.493628 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.493689 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:46.500891 (+ 7202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44417: BlockingRecv error: recv got EOF from 127.0.98.2:44417 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 21:07:46 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:46 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:46.619745 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:46.615345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:59393)
0927 21:07:46.615492 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:46.615514 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:46.615578 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:46.615697 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:46.615698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:46.615860 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:46.615908 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:46.615909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.616521 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.616522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:46.617042 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:46.617043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.617438 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.617438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:46.617503 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:46.617920 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:46.617925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:46.619648 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
W20260927 21:07:46.619793 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:46.615406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59393 (local address 127.0.98.62:36817)
0927 21:07:46.615472 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:46.615474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:46.615564 (+ 90us) server_negotiation.cc:407] Connection header received
0927 21:07:46.615628 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:46.615628 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:46.615638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:46.615676 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:46.615970 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.616453 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.617154 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:46.617348 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:46.617435 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:46.619729 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59393: BlockingRecv error: recv got EOF from 127.0.98.3:59393 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 21:07:47 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:47 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:47.371937 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:47.367682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:50699)
0927 21:07:47.367815 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:47.367828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:47.367880 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:47.368018 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:47.368019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:47.368179 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:47.368228 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:47.368229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.368843 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.368844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:47.369382 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:47.369384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.369708 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.369708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:47.369746 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:47.370176 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:47.370181 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:47.371877 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260927 21:07:47.372073 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:47.367748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50699 (local address 127.0.98.62:36817)
0927 21:07:47.367857 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:47.367859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:47.367869 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:47.367932 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:47.367933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:47.367942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:47.368002 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:47.368287 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.368782 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.369449 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.369646 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.369694 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:47.371965 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50699: BlockingRecv error: recv got EOF from 127.0.98.1:50699 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
Sep 27 21:07:47 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:47 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:47.507078 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:47.501281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:43187)
0927 21:07:47.501422 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:47.501433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:47.501499 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:47.501645 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:47.501646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:47.501851 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:47.501909 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:47.501912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.502951 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.502952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:47.503852 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:47.503855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.504444 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.504445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:47.504510 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:47.505098 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:47.505104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:47.507019 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
W20260927 21:07:47.507225 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:47.501342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43187 (local address 127.0.98.62:36817)
0927 21:07:47.501407 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:47.501409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:47.501480 (+ 71us) server_negotiation.cc:407] Connection header received
0927 21:07:47.501572 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:47.501573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:47.501588 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:47.501638 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:47.502015 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.502820 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.503927 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.504313 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.504379 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:47.507155 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43187: BlockingRecv error: recv got EOF from 127.0.98.2:43187 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 21:07:47 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:47 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:47.624610 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:47.620222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:51763)
0927 21:07:47.620367 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:47.620385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:47.620436 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:47.620563 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:47.620563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:47.620703 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:47.620751 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:47.620752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.621336 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.621337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:47.621835 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:47.621836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.622124 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.622124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:47.622162 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:47.622665 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:47.622671 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:47.624503 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260927 21:07:47.624660 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:47.620296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51763 (local address 127.0.98.62:36817)
0927 21:07:47.620352 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:47.620354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:47.620427 (+ 73us) server_negotiation.cc:407] Connection header received
0927 21:07:47.620476 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:47.620477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:47.620488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:47.620553 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:47.620809 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.621272 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.621899 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:47.622065 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:47.622112 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:47.624585 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51763: BlockingRecv error: recv got EOF from 127.0.98.3:51763 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 21:07:48 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:48 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:48.376710 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:48.372368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:57241)
0927 21:07:48.372507 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:48.372525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:48.372576 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:48.372731 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:48.372732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:48.372887 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:48.372935 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:48.372937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.373558 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.373558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:48.374062 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:48.374063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.374368 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.374369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:48.374406 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:48.374837 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:48.374841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:48.376613 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
W20260927 21:07:48.376821 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:48.372509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57241 (local address 127.0.98.62:36817)
0927 21:07:48.372620 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:48.372622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:48.372629 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:48.372648 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:48.372648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:48.372657 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:48.372712 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:48.373003 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.373496 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.374128 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.374309 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.374356 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:48.376746 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57241: BlockingRecv error: recv got EOF from 127.0.98.1:57241 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
Sep 27 21:07:48 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:48 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:48.511972 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:48.507519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:47087)
0927 21:07:48.507635 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:48.507646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:48.507713 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:48.507819 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:48.507820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:48.507973 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:48.508024 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:48.508025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.508784 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.508785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:48.509298 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:48.509300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.509611 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.509611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:48.509648 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:48.510132 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:48.510137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:48.511915 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
W20260927 21:07:48.512077 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:48.507589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47087 (local address 127.0.98.62:36817)
0927 21:07:48.507665 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:48.507666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:48.507698 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:07:48.507762 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:48.507763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:48.507774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:48.507813 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:48.508087 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.508716 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.509367 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.509550 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.509599 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:48.511996 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47087: BlockingRecv error: recv got EOF from 127.0.98.2:47087 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 21:07:48 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:48 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:48.629166 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:48.625034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:53779)
0927 21:07:48.625144 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:48.625154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:48.625210 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:48.625319 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:48.625319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:48.625499 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:48.625557 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:48.625559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.626175 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.626176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:48.626689 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:48.626691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.626992 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.626993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:48.627031 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:48.627458 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:48.627462 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:48.629110 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":82,"threads_started":1}
W20260927 21:07:48.629299 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:48.625074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53779 (local address 127.0.98.62:36817)
0927 21:07:48.625162 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:48.625166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:48.625205 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:07:48.625256 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:48.625257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:48.625269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:48.625315 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:48.625622 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.626114 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.626755 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:48.626929 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:48.626977 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:48.629215 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53779: BlockingRecv error: recv got EOF from 127.0.98.3:53779 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 21:07:49 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:49 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:49.381494 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:49.377192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:33349)
0927 21:07:49.377347 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:49.377357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:49.377416 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:49.377563 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:49.377563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:49.377716 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:49.377764 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:49.377765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.378374 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.378375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:49.378881 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:49.378882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.379262 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.379263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:49.379300 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:49.379713 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:49.379717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:49.381436 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
W20260927 21:07:49.381632 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:49.377282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33349 (local address 127.0.98.62:36817)
0927 21:07:49.377379 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:49.377380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:49.377400 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:07:49.377475 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:49.377476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:49.377485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:49.377546 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:49.377827 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.378316 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.378949 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.379199 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.379248 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:49.381537 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33349: BlockingRecv error: recv got EOF from 127.0.98.1:33349 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
Sep 27 21:07:49 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:49 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:49.516734 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:49.512499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:48293)
0927 21:07:49.512625 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:49.512635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:49.512696 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:49.512811 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:49.512811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:49.512964 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:49.513018 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:49.513019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.513619 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.513620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:49.514132 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:49.514133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.514433 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.514434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:49.514470 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:49.514886 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:49.514890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:49.516674 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260927 21:07:49.516808 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:49.512547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48293 (local address 127.0.98.62:36817)
0927 21:07:49.512612 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:49.512614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:49.512695 (+ 81us) server_negotiation.cc:407] Connection header received
0927 21:07:49.512739 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:49.512740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:49.512749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:49.512807 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:49.513072 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.513552 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.514198 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.514372 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.514420 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:49.516759 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48293: BlockingRecv error: recv got EOF from 127.0.98.2:48293 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 21:07:49 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:49 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:49.634018 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:49.629581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:54991)
0927 21:07:49.629703 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:49.629713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:49.629776 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:49.629893 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:49.629894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:49.630061 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:49.630126 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:49.630127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.630779 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.630779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:49.631514 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:49.631517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.631839 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.631839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:49.631877 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:49.632295 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:49.632300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:49.633957 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
W20260927 21:07:49.634133 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:49.629621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54991 (local address 127.0.98.62:36817)
0927 21:07:49.629696 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:49.629700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:49.629781 (+ 81us) server_negotiation.cc:407] Connection header received
0927 21:07:49.629819 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:49.629820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:49.629831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:49.629880 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:49.630208 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.630714 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.631588 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:49.631771 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:49.631832 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:49.634056 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54991: BlockingRecv error: recv got EOF from 127.0.98.3:54991 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 21:07:50 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:50 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:50.386327 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:50.381992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:42259)
0927 21:07:50.382132 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:50.382142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:50.382196 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:50.382308 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:50.382308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:50.382463 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:50.382519 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:50.382520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.383178 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.383179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:50.383695 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:50.383697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.384019 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.384020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:50.384060 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:50.384549 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:50.384555 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:50.386268 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
W20260927 21:07:50.386446 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:50.382024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42259 (local address 127.0.98.62:36817)
0927 21:07:50.382135 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:50.382137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:50.382183 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:07:50.382246 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:50.382247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:50.382256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:50.382297 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:50.382588 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.383096 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.383764 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.383946 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.384001 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:50.386339 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42259: BlockingRecv error: recv got EOF from 127.0.98.1:42259 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
Sep 27 21:07:50 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:50 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:50.522017 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:50.517251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:38157)
0927 21:07:50.517431 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:50.517443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:50.517533 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:50.517696 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:50.517697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:50.517874 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:50.517920 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:50.517922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.518589 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.518590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:50.519145 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:50.519146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.519501 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.519502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:50.519559 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:50.520017 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:50.520021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:50.521951 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
W20260927 21:07:50.522115 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:50.517339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38157 (local address 127.0.98.62:36817)
0927 21:07:50.517414 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:50.517417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:50.517517 (+ 100us) server_negotiation.cc:407] Connection header received
0927 21:07:50.517591 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:50.517592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:50.517605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:50.517683 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:50.518013 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.518506 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.519227 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.519426 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.519505 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:50.522046 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38157: BlockingRecv error: recv got EOF from 127.0.98.2:38157 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 21:07:50 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:50 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:50.638989 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:50.634471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:42145)
0927 21:07:50.634647 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:50.634657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:50.634713 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:50.634847 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:50.634848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:50.635014 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:50.635061 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:50.635063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.635736 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.635736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:50.636259 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:50.636261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.636559 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.636559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:50.636598 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:50.637037 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:50.637041 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:50.638921 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
W20260927 21:07:50.639125 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:50.634586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42145 (local address 127.0.98.62:36817)
0927 21:07:50.634658 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:50.634659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:50.634711 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:07:50.634768 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:50.634768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:50.634778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:50.634827 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:50.635188 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.635670 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.636326 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:50.636498 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:50.636547 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:50.639006 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42145: BlockingRecv error: recv got EOF from 127.0.98.3:42145 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 21:07:51 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:51 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:51.391338 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:51.386760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:55401)
0927 21:07:51.386915 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:51.386926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:51.386983 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:51.387219 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:51.387220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:51.387376 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:51.387424 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:51.387425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.388045 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.388045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:51.388561 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:51.388562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.388867 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.388868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:51.388905 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:51.389324 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:51.389328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:51.391280 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
W20260927 21:07:51.391467 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:51.386944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55401 (local address 127.0.98.62:36817)
0927 21:07:51.387105 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:51.387107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:51.387114 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:07:51.387133 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:51.387134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:51.387143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:51.387202 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:51.387491 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.387981 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.388627 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.388807 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.388855 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:51.391363 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55401: BlockingRecv error: recv got EOF from 127.0.98.1:55401 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
Sep 27 21:07:51 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:51 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:51.530762 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:51.522467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:38743)
0927 21:07:51.522621 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:51.522631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:51.522697 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:51.522818 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:51.522819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:51.522977 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:51.523026 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:51.523028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.523699 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.523699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:51.524211 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:51.524212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.524675 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.524676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:51.524720 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:51.529037 (+ 4317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:51.529041 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:51.530704 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260927 21:07:51.530871 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:51.522567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38743 (local address 127.0.98.62:36817)
0927 21:07:51.522613 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:51.522615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:51.522695 (+ 80us) server_negotiation.cc:407] Connection header received
0927 21:07:51.522742 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:51.522743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:51.522754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:51.522800 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:51.523121 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.523631 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.524331 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.524585 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.524662 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:51.530797 (+ 6135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38743: BlockingRecv error: recv got EOF from 127.0.98.2:38743 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 21:07:51 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:51 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:51.643743 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:51.639444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:43145)
0927 21:07:51.639576 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:51.639587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:51.639648 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:51.639762 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:51.639762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:51.639913 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:51.639959 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:51.639961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.640566 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.640566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:51.641091 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:51.641093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.641385 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.641385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:51.641423 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:51.641841 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:51.641846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:51.643637 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260927 21:07:51.643775 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:51.639495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43145 (local address 127.0.98.62:36817)
0927 21:07:51.639568 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:51.639570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:51.639648 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:07:51.639689 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:51.639690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:51.639702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:51.639754 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:51.640013 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.640489 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.641154 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:51.641325 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:51.641373 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:51.643710 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43145: BlockingRecv error: recv got EOF from 127.0.98.3:43145 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 21:07:52 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:52 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:52.396165 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:52.391728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:47289)
0927 21:07:52.391865 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:52.391875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:52.391923 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:52.392034 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:52.392035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:52.392194 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:52.392241 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:52.392242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.392918 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.392919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:52.393475 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:52.393477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.393844 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.393845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:52.393905 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:52.394349 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:52.394354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:52.396073 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
W20260927 21:07:52.396229 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:52.391792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47289 (local address 127.0.98.62:36817)
0927 21:07:52.391906 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:52.391908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:52.391916 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:52.391967 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:52.391967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:52.391978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:52.392064 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:52.392296 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.392800 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.393571 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.393771 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.393844 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:52.396161 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47289: BlockingRecv error: recv got EOF from 127.0.98.1:47289 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
Sep 27 21:07:52 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:52 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:52.535550 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:52.531328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:58985)
0927 21:07:52.531450 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:52.531461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:52.531526 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:52.531632 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:52.531633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:52.531788 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:52.531837 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:52.531838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.532453 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.532453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:52.533006 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:52.533007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.533313 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.533313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:52.533352 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:52.533777 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:52.533781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:52.535478 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
W20260927 21:07:52.535660 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:52.531398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58985 (local address 127.0.98.62:36817)
0927 21:07:52.531465 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:52.531467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:52.531527 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:07:52.531578 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:52.531578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:52.531588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:52.531623 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:52.531889 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.532381 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.533075 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.533252 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.533299 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:52.535566 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58985: BlockingRecv error: recv got EOF from 127.0.98.2:58985 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 21:07:52 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:52 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:52.648841 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:52.644221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:44419)
0927 21:07:52.644390 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:52.644400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:52.644482 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:52.644643 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:52.644644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:52.644809 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:52.644856 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:52.644858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.645527 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.645528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:52.646054 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:52.646055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.646598 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.646599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:52.646654 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:52.647143 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:52.647153 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:52.648787 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
W20260927 21:07:52.648947 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:52.644311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44419 (local address 127.0.98.62:36817)
0927 21:07:52.644375 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:52.644377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:52.644474 (+ 97us) server_negotiation.cc:407] Connection header received
0927 21:07:52.644543 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:52.644544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:52.644556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:52.644625 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:52.644929 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.645435 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.646136 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:52.646512 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:52.646582 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:52.648873 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44419: BlockingRecv error: recv got EOF from 127.0.98.3:44419 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 21:07:53 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:53 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:53.401062 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:53.396664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:55871)
0927 21:07:53.396912 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:53.396931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:53.396991 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:53.397120 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:53.397120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:53.397280 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:53.397329 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:53.397331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.397939 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.397940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:53.398468 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:53.398469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.398782 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.398782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:53.398819 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:53.399310 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:53.399315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:53.400993 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":68,"threads_started":1}
W20260927 21:07:53.401160 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:53.396710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55871 (local address 127.0.98.62:36817)
0927 21:07:53.396929 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:53.396931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:53.396975 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:07:53.397036 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:53.397036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:53.397046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:53.397099 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:53.397392 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.397877 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.398543 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.398721 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.398768 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:53.401084 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55871: BlockingRecv error: recv got EOF from 127.0.98.1:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":175,"threads_started":1}
Sep 27 21:07:53 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:53 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:53.540138 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:53.535933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:44911)
0927 21:07:53.536061 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:53.536079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:53.536127 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:53.536263 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:53.536263 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:53.536421 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:53.536468 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:53.536469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.537066 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.537067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:53.537580 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:53.537581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.537876 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.537877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:53.537915 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:53.538318 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:53.538322 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:53.540080 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
W20260927 21:07:53.540236 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:53.536083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44911 (local address 127.0.98.62:36817)
0927 21:07:53.536139 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:53.536141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:53.536170 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:07:53.536194 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:53.536194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:53.536204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:53.536243 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:53.536533 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.537007 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.537645 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.537819 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.537866 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:53.540170 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44911: BlockingRecv error: recv got EOF from 127.0.98.2:44911 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 21:07:53 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:53 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:53.653802 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:53.649348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:57831)
0927 21:07:53.649506 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:53.649519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:53.649582 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:53.649699 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:53.649699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:53.649857 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:53.649903 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:53.649904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.650521 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.650521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:53.651114 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:53.651117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.651421 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.651421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:53.651459 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:53.651931 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:53.651935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:53.653707 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
W20260927 21:07:53.653841 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:53.649406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57831 (local address 127.0.98.62:36817)
0927 21:07:53.649459 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:53.649461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:53.649585 (+ 124us) server_negotiation.cc:407] Connection header received
0927 21:07:53.649626 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:53.649626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:53.649638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:53.649694 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:53.649961 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.650440 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.651183 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:53.651362 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:53.651413 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:53.653782 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57831: BlockingRecv error: recv got EOF from 127.0.98.3:57831 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 21:07:54 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:54 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:54.406164 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:54.401606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:55113)
0927 21:07:54.401746 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:54.401758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:54.401826 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:54.401968 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:54.401969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:54.402135 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:54.402183 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:54.402186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.402813 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.402814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:54.403491 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:54.403493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.403825 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.403841 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:54.403903 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:54.404421 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:54.404426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:54.406107 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260927 21:07:54.406291 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:54.401674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55113 (local address 127.0.98.62:36817)
0927 21:07:54.401788 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:54.401789 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:54.401811 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:07:54.401878 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:54.401879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:54.401888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:54.401953 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:54.402241 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.402728 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.403555 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.403742 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.403795 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:54.406210 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55113: BlockingRecv error: recv got EOF from 127.0.98.1:55113 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
Sep 27 21:07:54 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:54 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:54.544961 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:54.540606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:37341)
0927 21:07:54.540772 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:54.540782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:54.540840 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:54.540960 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:54.540960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:54.541177 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:54.541238 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:54.541239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.541835 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.541836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:54.542359 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:54.542361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.542660 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.542660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:54.542698 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:54.543215 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:54.543221 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:54.544905 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
W20260927 21:07:54.545083 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:54.540651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37341 (local address 127.0.98.62:36817)
0927 21:07:54.540709 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:54.540712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:54.540843 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:07:54.540885 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:54.540885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:54.540894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:54.540946 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:54.541304 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.541772 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.542424 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.542601 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.542648 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:54.544987 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37341: BlockingRecv error: recv got EOF from 127.0.98.2:37341 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 21:07:54 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:54 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:54.658644 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:54.654251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.3:34221)
0927 21:07:54.654419 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:54.654433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:54.654498 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:54.654625 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:54.654626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:54.654784 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:54.654831 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:54.654832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.655450 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.655451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:54.655962 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:54.655964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.656268 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.656269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:54.656307 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:54.656795 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:54.656800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:54.658552 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
W20260927 21:07:54.658682 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:54.654334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34221 (local address 127.0.98.62:36817)
0927 21:07:54.654403 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:54.654405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:54.654503 (+ 98us) server_negotiation.cc:407] Connection header received
0927 21:07:54.654555 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:54.654555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:54.654569 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:54.654620 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:54.654891 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.655390 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.656036 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:54.656209 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:54.656256 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:54.658625 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34221: BlockingRecv error: recv got EOF from 127.0.98.3:34221 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 21:07:55 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:55 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:55.411170 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:55.406844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.1:60035)
0927 21:07:55.406992 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:55.407002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:55.407051 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:55.407287 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:55.407288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:55.407444 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:55.407491 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:55.407492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.408124 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.408125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:55.408625 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:55.408626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.408941 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.408942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:55.408978 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.409406 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:55.409410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:55.411049 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
W20260927 21:07:55.411285 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:55.406911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60035 (local address 127.0.98.62:36817)
0927 21:07:55.407052 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:55.407054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:55.407121 (+ 67us) server_negotiation.cc:407] Connection header received
0927 21:07:55.407146 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:55.407147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:55.407160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:55.407195 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:55.407557 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.408060 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.408693 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.408877 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.408926 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.411185 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60035: BlockingRecv error: recv got EOF from 127.0.98.1:60035 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
Sep 27 21:07:55 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:07:55 dist-test-slave-8bgs krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:07:55.550181 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:55.545489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36817 (local address 127.0.98.2:58735)
0927 21:07:55.545655 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:55.545669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:55.545741 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:55.545953 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:55.545955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:55.546201 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:55.546249 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:55.546251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.546882 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.546882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:55.547634 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:55.547637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.547954 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.547955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:55.547991 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.548396 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:55.548401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:55.550112 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36817: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260927 21:07:55.550299 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:55.545644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58735 (local address 127.0.98.62:36817)
0927 21:07:55.545708 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:55.545710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:55.545721 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:07:55.545849 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:55.545851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:55.545865 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:55.545932 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:55.546324 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.546814 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.547715 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.547894 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.547943 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.550224 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58735: BlockingRecv error: recv got EOF from 127.0.98.2:58735 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:07:55.577590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9110
I20260927 21:07:55.582448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9262
I20260927 21:07:55.586731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9394
I20260927 21:07:55.590714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9040
2026-09-27T21:07:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17386 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T21:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:55Z Disabled control of system clock
I20260927 21:07:55.617825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44227
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:07:55.696472 9590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:55.696661 9590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:55.696681 9590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:55.697849 9590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:55.697889 9590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:55.697902 9590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:55.697914 9590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:55.697962 9590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:55.699280 9590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:55.699517 9590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:55.699796 9590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:55.704613 9595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:55.704658 9590 server_base.cc:1034] running on GCE node
W20260927 21:07:55.704630 9598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:55.704639 9596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:55.705127 9590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:55.705431 9590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:55.706598 9590 hybrid_clock.cc:648] HybridClock initialized: now 1790543275706586 us; error 43 us; skew 500 ppm
I20260927 21:07:55.707865 9590 webserver.cc:460] Webserver started at http://127.0.98.62:46009/ using document root <none> and password file <none>
I20260927 21:07:55.708120 9590 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:55.708189 9590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:55.708324 9590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:55.709355 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "e09baafa93ae475c871f8052cdfd6239"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "c27db6998a3bf908b83b736d3558dd20"
server_key_iv: "f1c91816b162e1178ade711c6a93afa0"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.709726 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "e09baafa93ae475c871f8052cdfd6239"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "c27db6998a3bf908b83b736d3558dd20"
server_key_iv: "f1c91816b162e1178ade711c6a93afa0"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.711030 9590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:55.711797 9604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:55.711969 9590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:55.712059 9590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "e09baafa93ae475c871f8052cdfd6239"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "c27db6998a3bf908b83b736d3558dd20"
server_key_iv: "f1c91816b162e1178ade711c6a93afa0"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.712143 9590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:55.720005 9590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:55.720271 9590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:55.720409 9590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:55.724182 9590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44227
I20260927 21:07:55.724210 9656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44227 every 8 connection(s)
I20260927 21:07:55.724529 9590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:55.725118 9657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:55.727632 9657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239: Bootstrap starting.
I20260927 21:07:55.728186 9657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:55.728472 9657 log.cc:824] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:55.729092 9657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239: No bootstrap required, opened a new log
I20260927 21:07:55.730588 9657 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09baafa93ae475c871f8052cdfd6239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260927 21:07:55.730713 9657 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:55.730759 9657 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09baafa93ae475c871f8052cdfd6239, State: Initialized, Role: FOLLOWER
I20260927 21:07:55.730898 9657 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09baafa93ae475c871f8052cdfd6239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260927 21:07:55.730975 9657 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:55.730998 9657 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:55.731062 9657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:55.731593 9657 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09baafa93ae475c871f8052cdfd6239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260927 21:07:55.731696 9657 leader_election.cc:302] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09baafa93ae475c871f8052cdfd6239; no voters:
I20260927 21:07:55.731884 9657 leader_election.cc:288] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:55.731907 9660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:55.731986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9590
I20260927 21:07:55.732012 9660 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [term 1 LEADER]: Becoming Leader. State: Replica: e09baafa93ae475c871f8052cdfd6239, State: Running, Role: LEADER
I20260927 21:07:55.732085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:55.732096 9660 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09baafa93ae475c871f8052cdfd6239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260927 21:07:55.732125 9657 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:55.732267 392 external_mini_cluster.cc:1428] Setting key e8579cb3a011d322921159471f72f70a
I20260927 21:07:55.732383 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09baafa93ae475c871f8052cdfd6239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09baafa93ae475c871f8052cdfd6239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } } }
I20260927 21:07:55.732458 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:55.732547 9662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09baafa93ae475c871f8052cdfd6239. Latest consensus state: current_term: 1 leader_uuid: "e09baafa93ae475c871f8052cdfd6239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09baafa93ae475c871f8052cdfd6239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } } }
I20260927 21:07:55.732625 9662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:55.732964 9665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:55.733506 9665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:55.734956 9665 catalog_manager.cc:1347] Generated new cluster ID: c61e10e174b9419c8494ac65eb22a5cf
I20260927 21:07:55.735015 9665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:55.738528 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:55.732776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.0.98.62:44227)
0927 21:07:55.733251 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:55.733260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:55.733295 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:07:55.733418 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:55.733426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:55.733502 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:55.733931 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:55.734028 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.735361 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.736421 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.737044 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.737173 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.737849 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:55.737852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:55.737853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:55.738051 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:55.738092 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:55.738107 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:55.738177 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:55.738248 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:07:55.738322 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":71,"threads_started":1}
I20260927 21:07:55.741499 9665 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:55.741966 9665 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:55.759049 9665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09baafa93ae475c871f8052cdfd6239: Generated new TSK 0
I20260927 21:07:55.759244 9665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:55.765585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:07:55.845432 9681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:55.845633 9681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:55.845654 9681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:55.846786 9681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:55.846827 9681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:55.846850 9681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:55.848258 9681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:55.848507 9681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:55.848794 9681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:55.854452 9686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:55.854462 9687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:55.854463 9689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:55.854727 9681 server_base.cc:1034] running on GCE node
I20260927 21:07:55.854933 9681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:55.855232 9681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:55.856385 9681 hybrid_clock.cc:648] HybridClock initialized: now 1790543275856370 us; error 33 us; skew 500 ppm
I20260927 21:07:55.857545 9681 webserver.cc:460] Webserver started at http://127.0.98.1:33035/ using document root <none> and password file <none>
I20260927 21:07:55.857781 9681 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:55.857826 9681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:55.857954 9681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:55.859002 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "e56ff917b94748a1a0d154e4ead0f9db"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "e423a3536bd676dce3da9f159b57f14a"
server_key_iv: "7bec2629317769b02081bbf1bde602e6"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.859417 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e56ff917b94748a1a0d154e4ead0f9db"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "e423a3536bd676dce3da9f159b57f14a"
server_key_iv: "7bec2629317769b02081bbf1bde602e6"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.860759 9681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:55.861490 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:55.861686 9681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:55.861773 9681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "e56ff917b94748a1a0d154e4ead0f9db"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "e423a3536bd676dce3da9f159b57f14a"
server_key_iv: "7bec2629317769b02081bbf1bde602e6"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.861840 9681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:55.876082 9681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:55.876358 9681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:55.876505 9681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:55.876720 9681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:55.877025 9681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:55.877058 9681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:55.877079 9681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:55.877137 9681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:55.882822 9681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45339
I20260927 21:07:55.882871 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45339 every 8 connection(s)
I20260927 21:07:55.883191 9681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:55.887861 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:55.883846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38839 (local address 127.0.98.62:44227)
0927 21:07:55.883919 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:55.883922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:55.884160 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:07:55.884604 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:55.884605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:55.884618 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:55.884669 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:55.885254 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.885750 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.886656 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.886832 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.886880 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.887653 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:55.887655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:55.887655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:55.887661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:55.887673 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:55.887677 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:55.887721 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:55.887781 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:07:55.887807 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:07:55.888051 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:55.883826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44227 (local address 127.0.98.1:38839)
0927 21:07:55.884037 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:55.884070 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:55.884526 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:55.884722 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:55.884725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:55.884732 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:55.885189 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:55.885193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.885829 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.885830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:55.886586 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:55.886588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:55.886902 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:55.886902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:55.887012 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:55.887534 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:55.887535 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:55.887561 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:55.887582 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:55.887730 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:55.887733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:55.887735 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:55.887784 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:07:55.887856 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260927 21:07:55.888520 9809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44227
I20260927 21:07:55.888594 9809 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:55.888723 9809 heartbeater.cc:511] Master 127.0.98.62:44227 requested a full tablet report, sending...
I20260927 21:07:55.889145 9621 ts_manager.cc:194] Registered new tserver with Master: e56ff917b94748a1a0d154e4ead0f9db (127.0.98.1:45339)
I20260927 21:07:55.889729 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38839
I20260927 21:07:55.890184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9681
I20260927 21:07:55.890280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:55.890477 392 external_mini_cluster.cc:1428] Setting key ce09897941fc5cf6c9f0b53fb17ddb60
I20260927 21:07:55.891300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:07:55.970466 9812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:55.970695 9812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:55.970728 9812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:55.971915 9812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:55.971985 9812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:55.972028 9812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:55.973345 9812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:55.973589 9812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:55.973860 9812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:55.978674 9820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:55.978654 9817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:55.978650 9818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:55.978706 9812 server_base.cc:1034] running on GCE node
I20260927 21:07:55.979053 9812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:55.979334 9812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:55.980476 9812 hybrid_clock.cc:648] HybridClock initialized: now 1790543275980462 us; error 30 us; skew 500 ppm
I20260927 21:07:55.981600 9812 webserver.cc:460] Webserver started at http://127.0.98.2:40033/ using document root <none> and password file <none>
I20260927 21:07:55.981781 9812 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:55.981820 9812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:55.981905 9812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:55.982856 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb1685394dea4d3fafd5022b8f19f700"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "a6e26f0cd38e2e9b73378928cd7d891b"
server_key_iv: "555778388d12dec9c110e807d0dea258"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.983218 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb1685394dea4d3fafd5022b8f19f700"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "a6e26f0cd38e2e9b73378928cd7d891b"
server_key_iv: "555778388d12dec9c110e807d0dea258"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.984397 9812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:07:55.985016 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:55.985185 9812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:55.985287 9812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "fb1685394dea4d3fafd5022b8f19f700"
format_stamp: "Formatted at 2026-09-27 21:07:55 on dist-test-slave-8bgs"
server_key: "a6e26f0cd38e2e9b73378928cd7d891b"
server_key_iv: "555778388d12dec9c110e807d0dea258"
server_key_version: "encryptionkey@0"
I20260927 21:07:55.985356 9812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:56.010984 9812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:56.011317 9812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:56.011469 9812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:56.011670 9812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:56.011942 9812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:56.011974 9812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.011998 9812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:56.012012 9812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.017447 9812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44861
I20260927 21:07:56.017568 9939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44861 every 8 connection(s)
I20260927 21:07:56.017851 9812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:56.022779 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.018601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55181 (local address 127.0.98.62:44227)
0927 21:07:56.018645 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.018649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.018903 (+ 254us) server_negotiation.cc:407] Connection header received
0927 21:07:56.019462 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.019462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.019472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.019533 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.020170 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.020665 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.021590 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.021765 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.021815 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:56.022563 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:56.022564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:56.022564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:56.022568 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:56.022579 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:56.022583 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:56.022624 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:56.022702 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:07:56.022722 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:07:56.022984 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.018576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44227 (local address 127.0.98.2:55181)
0927 21:07:56.018774 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:56.018814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:56.019372 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:56.019588 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:56.019592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:56.019600 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:56.020095 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:56.020099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.020748 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.020748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:56.021517 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:56.021518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.021837 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.021838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:56.021943 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:56.022437 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:56.022437 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:56.022465 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:56.022489 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:56.022636 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:56.022639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:56.022642 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:56.022692 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:07:56.022770 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260927 21:07:56.023538 9940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44227
I20260927 21:07:56.023636 9940 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:56.023820 9940 heartbeater.cc:511] Master 127.0.98.62:44227 requested a full tablet report, sending...
I20260927 21:07:56.024200 9621 ts_manager.cc:194] Registered new tserver with Master: fb1685394dea4d3fafd5022b8f19f700 (127.0.98.2:44861)
I20260927 21:07:56.024525 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55181
I20260927 21:07:56.025724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9812
I20260927 21:07:56.025830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:56.026036 392 external_mini_cluster.cc:1428] Setting key 8cc84526f9a404b1591da302e757a331
I20260927 21:07:56.026822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--builtin_ntp_servers=127.0.98.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:07:56.107905 9943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:56.108100 9943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:56.108121 9943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:56.109344 9943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:56.109385 9943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:56.109409 9943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:56.110716 9943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:56.110944 9943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:56.111249 9943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:56.116045 9949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.116019 9951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.116009 9948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:56.116223 9943 server_base.cc:1034] running on GCE node
I20260927 21:07:56.116437 9943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:56.116665 9943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:56.117820 9943 hybrid_clock.cc:648] HybridClock initialized: now 1790543276117800 us; error 32 us; skew 500 ppm
I20260927 21:07:56.119004 9943 webserver.cc:460] Webserver started at http://127.0.98.3:37521/ using document root <none> and password file <none>
I20260927 21:07:56.119270 9943 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:56.119342 9943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:56.119494 9943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:56.120514 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "011e7ea8dcb94718ad4657448b7b4485"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "c1fbca5d7af8542afba457a6a1fec212"
server_key_iv: "2364dc69d9a62faf6b58a71e21600492"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.120859 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "011e7ea8dcb94718ad4657448b7b4485"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "c1fbca5d7af8542afba457a6a1fec212"
server_key_iv: "2364dc69d9a62faf6b58a71e21600492"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.122112 9943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:56.122780 9957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.122941 9943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:56.123042 9943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "011e7ea8dcb94718ad4657448b7b4485"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "c1fbca5d7af8542afba457a6a1fec212"
server_key_iv: "2364dc69d9a62faf6b58a71e21600492"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.123150 9943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:56.140658 9943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:56.140995 9943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:56.141158 9943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:56.141396 9943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:56.141767 9943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:56.141822 9943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.141856 9943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:56.141886 9943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.147624 9943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45243
I20260927 21:07:56.147694 10070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45243 every 8 connection(s)
I20260927 21:07:56.148018 9943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:56.151774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 9943
I20260927 21:07:56.151888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:56.152112 392 external_mini_cluster.cc:1428] Setting key ebd1e07750d27e00d18e7d8c8bd4e838
I20260927 21:07:56.153657 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.148714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33775 (local address 127.0.98.62:44227)
0927 21:07:56.148766 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.148768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.148987 (+ 219us) server_negotiation.cc:407] Connection header received
0927 21:07:56.149523 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.149526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.149538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.149592 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.150279 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.150766 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.151981 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.152294 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.152341 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:56.153375 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:56.153377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:56.153377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:56.153381 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:56.153395 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:56.153399 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:56.153479 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:56.153555 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 21:07:56.153582 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:56.153877 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.148675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44227 (local address 127.0.98.3:33775)
0927 21:07:56.148845 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:56.148884 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:56.149412 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:56.149662 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:56.149666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:56.149676 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:56.150166 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:56.150171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.150860 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.150861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:56.151859 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:56.151862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:56.152434 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:56.152435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:56.152591 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:56.153219 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:56.153220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:56.153260 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:56.153283 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:56.153481 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:56.153486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:56.153490 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:56.153559 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:07:56.153658 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260927 21:07:56.154302 10071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44227
I20260927 21:07:56.154374 10071 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:56.154567 10071 heartbeater.cc:511] Master 127.0.98.62:44227 requested a full tablet report, sending...
I20260927 21:07:56.154908 9621 ts_manager.cc:194] Registered new tserver with Master: 011e7ea8dcb94718ad4657448b7b4485 (127.0.98.3:45243)
I20260927 21:07:56.155296 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33775
I20260927 21:07:56.163872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:07:56.165695 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:56.165145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.98.62:44227)
0927 21:07:56.165218 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.165220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.165309 (+ 89us) server_negotiation.cc:407] Connection header received
0927 21:07:56.165384 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.165385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.165403 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.165464 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.165612 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45942: BlockingRecv error: recv got EOF from 127.0.0.1:45942 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:07:56.165737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9681
I20260927 21:07:56.170236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9812
I20260927 21:07:56.174396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9943
I20260927 21:07:56.178419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 9590
2026-09-27T21:07:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (586 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T21:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:56Z Disabled control of system clock
I20260927 21:07:56.204119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45845
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:07:56.283447 10082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:56.283641 10082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:56.283663 10082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:56.284812 10082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:56.284853 10082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:56.284866 10082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:56.284879 10082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:56.284947 10082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:56.286293 10082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:56.286521 10082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:56.286789 10082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:56.291440 10088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.291477 10087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.291460 10090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:56.291677 10082 server_base.cc:1034] running on GCE node
I20260927 21:07:56.291884 10082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:56.292148 10082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:56.293285 10082 hybrid_clock.cc:648] HybridClock initialized: now 1790543276293271 us; error 29 us; skew 500 ppm
I20260927 21:07:56.294420 10082 webserver.cc:460] Webserver started at http://127.0.98.62:45479/ using document root <none> and password file <none>
I20260927 21:07:56.294627 10082 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:56.294668 10082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:56.294757 10082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:56.295799 10082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "ae13c8f49f9c4be3927ccd75e5df1072"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "d8d474d5b195d21aa0ce99596904bec0"
server_key_iv: "67b2917ada7498d1b38ac0fa84e64d6b"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.296096 10082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae13c8f49f9c4be3927ccd75e5df1072"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "d8d474d5b195d21aa0ce99596904bec0"
server_key_iv: "67b2917ada7498d1b38ac0fa84e64d6b"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.297287 10082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:07:56.297941 10096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.298092 10082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:56.298159 10082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "ae13c8f49f9c4be3927ccd75e5df1072"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "d8d474d5b195d21aa0ce99596904bec0"
server_key_iv: "67b2917ada7498d1b38ac0fa84e64d6b"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.298209 10082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:56.301367 10082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:56.301589 10082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:56.301682 10082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:56.305285 10082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45845
I20260927 21:07:56.305330 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45845 every 8 connection(s)
I20260927 21:07:56.305627 10082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:56.306248 10149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:56.307932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10082
I20260927 21:07:56.308009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:56.308202 392 external_mini_cluster.cc:1428] Setting key f2fe5eff9bbff8308ae4b373432e94ea
I20260927 21:07:56.309137 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072: Bootstrap starting.
I20260927 21:07:56.309675 10149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:56.310017 10149 log.cc:824] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:56.310727 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.308709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.62:45845)
0927 21:07:56.308910 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.308925 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.308948 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:07:56.309380 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.309386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.309427 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.309576 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.310287 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:56.310291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:56.310291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:56.310295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:56.310346 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:56.310355 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:56.310426 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:56.310515 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 21:07:56.310571 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260927 21:07:56.310730 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072: No bootstrap required, opened a new log
I20260927 21:07:56.312261 10149 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260927 21:07:56.312409 10149 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:56.312459 10149 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae13c8f49f9c4be3927ccd75e5df1072, State: Initialized, Role: FOLLOWER
I20260927 21:07:56.312575 10149 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260927 21:07:56.312664 10149 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:56.312707 10149 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:56.312754 10149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:56.313491 10149 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260927 21:07:56.313616 10149 leader_election.cc:302] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae13c8f49f9c4be3927ccd75e5df1072; no voters:
I20260927 21:07:56.313784 10149 leader_election.cc:288] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:56.313827 10154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:56.313980 10154 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [term 1 LEADER]: Becoming Leader. State: Replica: ae13c8f49f9c4be3927ccd75e5df1072, State: Running, Role: LEADER
I20260927 21:07:56.314040 10149 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:56.314126 10154 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260927 21:07:56.314472 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } } }
I20260927 21:07:56.314491 10156 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae13c8f49f9c4be3927ccd75e5df1072. Latest consensus state: current_term: 1 leader_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae13c8f49f9c4be3927ccd75e5df1072" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } } }
I20260927 21:07:56.314620 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:56.314623 10156 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072 [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:56.315230 10163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:56.315719 10163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:56.317018 10163 catalog_manager.cc:1347] Generated new cluster ID: 4796ee8906fc46f1a72b460e60ee30ca
I20260927 21:07:56.317066 10163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:56.349010 10163 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:56.349202 10163 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:56.353911 10163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae13c8f49f9c4be3927ccd75e5df1072: Generated new TSK 0
I20260927 21:07:56.354089 10163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:07:56.365582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45845
--builtin_ntp_servers=127.0.98.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:07:56.446480 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:56.446676 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:56.446697 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:56.447829 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:56.447865 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:56.447889 10173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:56.449150 10173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:56.449324 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:56.449563 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:56.454327 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.454320 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.454319 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:56.454595 10173 server_base.cc:1034] running on GCE node
I20260927 21:07:56.454819 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:56.455053 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:56.456234 10173 hybrid_clock.cc:648] HybridClock initialized: now 1790543276456222 us; error 27 us; skew 500 ppm
I20260927 21:07:56.457397 10173 webserver.cc:460] Webserver started at http://127.0.98.1:43059/ using document root <none> and password file <none>
I20260927 21:07:56.457629 10173 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:56.457674 10173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:56.457836 10173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:56.458848 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "c866d423e0b74c6da1e6d613bc43bdef"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "7bb9bbbf4fd2ddf12d33a4c1c648679d"
server_key_iv: "437540dc880b6c5c4b60764c79cc21b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.459220 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c866d423e0b74c6da1e6d613bc43bdef"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "7bb9bbbf4fd2ddf12d33a4c1c648679d"
server_key_iv: "437540dc880b6c5c4b60764c79cc21b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.460546 10173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:56.461189 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.461345 10173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:56.461429 10173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "c866d423e0b74c6da1e6d613bc43bdef"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "7bb9bbbf4fd2ddf12d33a4c1c648679d"
server_key_iv: "437540dc880b6c5c4b60764c79cc21b7"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.461503 10173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:56.464818 10173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:56.465056 10173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:56.465194 10173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:56.465402 10173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:56.465693 10173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:56.465725 10173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.465785 10173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:56.465811 10173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.471549 10173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35731
I20260927 21:07:56.471597 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35731 every 8 connection(s)
I20260927 21:07:56.471923 10173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:07:56.474759 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.472646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54221 (local address 127.0.98.62:45845)
0927 21:07:56.472687 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.472689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.472927 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:07:56.473365 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.473366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.473371 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.473429 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.474569 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:56.474569 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:56.474570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:56.474572 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:56.474581 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:56.474584 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:56.474623 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:56.474697 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:07:56.474708 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:07:56.474973 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.472605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45845 (local address 127.0.98.1:54221)
0927 21:07:56.472797 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:56.472837 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:56.473284 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:56.473479 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:56.473482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:56.473490 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:56.474448 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:56.474448 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:56.474475 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:56.474501 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:56.474623 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:56.474625 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:56.474627 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:56.474695 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:07:56.474746 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260927 21:07:56.475427 10301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45845
I20260927 21:07:56.475505 10301 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:56.475664 10301 heartbeater.cc:511] Master 127.0.98.62:45845 requested a full tablet report, sending...
I20260927 21:07:56.476145 10113 ts_manager.cc:194] Registered new tserver with Master: c866d423e0b74c6da1e6d613bc43bdef (127.0.98.1:35731)
I20260927 21:07:56.480204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10173
I20260927 21:07:56.480286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:56.480484 392 external_mini_cluster.cc:1428] Setting key 5193919565f8f7db07198eebec624db7
I20260927 21:07:56.481366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45845
--builtin_ntp_servers=127.0.98.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:07:56.560495 10304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:56.560683 10304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:56.560704 10304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:56.561842 10304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:56.561880 10304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:56.561931 10304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:56.563257 10304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:56.563485 10304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:56.563797 10304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:56.568573 10309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.568627 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:56.568645 10304 server_base.cc:1034] running on GCE node
W20260927 21:07:56.568581 10312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:56.568944 10304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:56.569206 10304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:56.570345 10304 hybrid_clock.cc:648] HybridClock initialized: now 1790543276570319 us; error 46 us; skew 500 ppm
I20260927 21:07:56.575311 10304 webserver.cc:460] Webserver started at http://127.0.98.2:42409/ using document root <none> and password file <none>
I20260927 21:07:56.575531 10304 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:56.575577 10304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:56.575713 10304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:56.576738 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "b13a5654e8c2452b96a2d5bd555401c6"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "5ff4978bdbd25a1dbe65948d29f52cbf"
server_key_iv: "6fb906ac85201edaf3cbe9243d66145a"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.577085 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b13a5654e8c2452b96a2d5bd555401c6"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "5ff4978bdbd25a1dbe65948d29f52cbf"
server_key_iv: "6fb906ac85201edaf3cbe9243d66145a"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.578337 10304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:07:56.579010 10318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.579192 10304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:56.579286 10304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "b13a5654e8c2452b96a2d5bd555401c6"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "5ff4978bdbd25a1dbe65948d29f52cbf"
server_key_iv: "6fb906ac85201edaf3cbe9243d66145a"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.579363 10304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:56.582542 10304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:56.582795 10304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:56.582935 10304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:56.583194 10304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:56.583495 10304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:56.583546 10304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.583585 10304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:56.583611 10304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.589267 10304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35951
I20260927 21:07:56.589308 10431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35951 every 8 connection(s)
I20260927 21:07:56.589597 10304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:07:56.592338 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.590206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49397 (local address 127.0.98.62:45845)
0927 21:07:56.590245 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.590248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.590455 (+ 207us) server_negotiation.cc:407] Connection header received
0927 21:07:56.590952 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.590954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.590962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.591021 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.592109 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:56.592110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:56.592111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:56.592115 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:56.592131 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:56.592136 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:56.592184 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:56.592255 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:07:56.592269 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:07:56.592530 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.590183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45845 (local address 127.0.98.2:49397)
0927 21:07:56.590338 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:56.590369 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:56.590833 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:56.591101 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:56.591106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:56.591115 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:56.591960 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:56.591961 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:56.591989 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:56.592015 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:56.592191 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:56.592196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:56.592199 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:56.592265 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:07:56.592322 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20260927 21:07:56.592969 10432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45845
I20260927 21:07:56.593060 10432 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:56.593170 10432 heartbeater.cc:511] Master 127.0.98.62:45845 requested a full tablet report, sending...
I20260927 21:07:56.593482 10113 ts_manager.cc:194] Registered new tserver with Master: b13a5654e8c2452b96a2d5bd555401c6 (127.0.98.2:35951)
I20260927 21:07:56.595511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10304
I20260927 21:07:56.595605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:56.595750 392 external_mini_cluster.cc:1428] Setting key 75debda1f1f87037944fbea703df0695
I20260927 21:07:56.596562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45845
--builtin_ntp_servers=127.0.98.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:07:56.675777 10435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:56.676008 10435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:56.676044 10435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:56.677201 10435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:56.677265 10435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:56.677306 10435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:56.678648 10435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:56.678905 10435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:56.679231 10435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:56.684085 10443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.684127 10441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:56.684129 10440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:56.684434 10435 server_base.cc:1034] running on GCE node
I20260927 21:07:56.684674 10435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:56.684937 10435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:56.686122 10435 hybrid_clock.cc:648] HybridClock initialized: now 1790543276686111 us; error 25 us; skew 500 ppm
I20260927 21:07:56.687369 10435 webserver.cc:460] Webserver started at http://127.0.98.3:35435/ using document root <none> and password file <none>
I20260927 21:07:56.687604 10435 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:56.687674 10435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:56.687808 10435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:56.688935 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "0ac18b7048c842ac93296b0a90ced7cf"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "ee7d68dc46809d8fad50470f006a6656"
server_key_iv: "87b575cf99591a160c04eaf6f31b3749"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.689311 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0ac18b7048c842ac93296b0a90ced7cf"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "ee7d68dc46809d8fad50470f006a6656"
server_key_iv: "87b575cf99591a160c04eaf6f31b3749"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.690606 10435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:56.691368 10449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.691526 10435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:56.691629 10435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "0ac18b7048c842ac93296b0a90ced7cf"
format_stamp: "Formatted at 2026-09-27 21:07:56 on dist-test-slave-8bgs"
server_key: "ee7d68dc46809d8fad50470f006a6656"
server_key_iv: "87b575cf99591a160c04eaf6f31b3749"
server_key_version: "encryptionkey@0"
I20260927 21:07:56.691705 10435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:56.694989 10435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:56.695273 10435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:56.695420 10435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:56.695636 10435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:56.695938 10435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:56.695992 10435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.696033 10435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:56.696058 10435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:56.701922 10435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38971
I20260927 21:07:56.701969 10562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38971 every 8 connection(s)
I20260927 21:07:56.702312 10435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:07:56.705051 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.703020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52973 (local address 127.0.98.62:45845)
0927 21:07:56.703062 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.703063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.703350 (+ 287us) server_negotiation.cc:407] Connection header received
0927 21:07:56.703774 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.703776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.703781 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.703837 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.704861 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:56.704862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:56.704863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:07:56.704866 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:56.704879 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:07:56.704883 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:56.704928 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:56.704988 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:07:56.705001 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:07:56.705255 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:56.703024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45845 (local address 127.0.98.3:52973)
0927 21:07:56.703216 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:56.703262 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:56.703693 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:56.703886 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:56.703889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:56.703896 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:56.704728 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:07:56.704728 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:56.704755 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:07:56.704783 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:56.704927 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:56.704929 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:56.704931 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:07:56.704987 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:07:56.705032 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260927 21:07:56.705675 10563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45845
I20260927 21:07:56.705763 10563 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:56.705875 10563 heartbeater.cc:511] Master 127.0.98.62:45845 requested a full tablet report, sending...
I20260927 21:07:56.706195 10113 ts_manager.cc:194] Registered new tserver with Master: 0ac18b7048c842ac93296b0a90ced7cf (127.0.98.3:38971)
I20260927 21:07:56.710879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10435
I20260927 21:07:56.710953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:56.711197 392 external_mini_cluster.cc:1428] Setting key c45742f66caab7a5877a6d252a404c7c
I20260927 21:07:56.712076 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:07:56.713901 10152 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:07:56.713337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.98.62:45845)
0927 21:07:56.713435 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:56.713436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:56.713544 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:07:56.713622 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:56.713623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:56.713628 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:56.713702 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:56.713836 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41216: BlockingRecv error: recv got EOF from 127.0.0.1:41216 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:07:56.713984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10173
I20260927 21:07:56.718850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10304
I20260927 21:07:56.723131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10435
I20260927 21:07:56.726948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10082
2026-09-27T21:07:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (548 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:56 dist-test-slave-8bgs krb5kdc[10573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:56 dist-test-slave-8bgs krb5kdc[10573](info): set up 2 sockets
Sep 27 21:07:56 dist-test-slave-8bgs krb5kdc[10573](info): commencing operation
krb5kdc: starting...
W20260927 21:07:57.743906 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:57 dist-test-slave-8bgs krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:07:57.854737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34351
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:57.932319 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:57.932564 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:57.932602 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:57.933737 10586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:07:57.933808 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:57.933831 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:57.933852 10586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:07:57.933883 10586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:07:57.935360 10586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:57.935642 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:57.935866 10586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:57.940565 10591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:57.940614 10592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:57.940759 10586 server_base.cc:1034] running on GCE node
W20260927 21:07:57.940572 10594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:57.941067 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:57.941351 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:57.942512 10586 hybrid_clock.cc:648] HybridClock initialized: now 1790543277942506 us; error 35 us; skew 500 ppm
Sep 27 21:07:57 dist-test-slave-8bgs krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:57.944849 10586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:07:57.945274 10586 webserver.cc:460] Webserver started at http://127.0.98.62:43921/ using document root <none> and password file <none>
I20260927 21:07:57.945503 10586 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:57.945547 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:57.945683 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:57.946683 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "de5e8671169b49c9baebabb2f359be2a"
format_stamp: "Formatted at 2026-09-27 21:07:57 on dist-test-slave-8bgs"
server_key: "53e86edc69f975f6ad369f85cc0844aa"
server_key_iv: "a02a68201fda04ad106a9273fd43f422"
server_key_version: "encryptionkey@0"
I20260927 21:07:57.947031 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "de5e8671169b49c9baebabb2f359be2a"
format_stamp: "Formatted at 2026-09-27 21:07:57 on dist-test-slave-8bgs"
server_key: "53e86edc69f975f6ad369f85cc0844aa"
server_key_iv: "a02a68201fda04ad106a9273fd43f422"
server_key_version: "encryptionkey@0"
I20260927 21:07:57.948354 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:07:57.949008 10601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:57.949172 10586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:57.949266 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "de5e8671169b49c9baebabb2f359be2a"
format_stamp: "Formatted at 2026-09-27 21:07:57 on dist-test-slave-8bgs"
server_key: "53e86edc69f975f6ad369f85cc0844aa"
server_key_iv: "a02a68201fda04ad106a9273fd43f422"
server_key_version: "encryptionkey@0"
I20260927 21:07:57.949342 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:57.960000 10586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:57.961627 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:57.961776 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:57.965440 10586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34351
I20260927 21:07:57.965466 10653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34351 every 8 connection(s)
I20260927 21:07:57.965791 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:07:57.966733 10654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:57.968807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10586
I20260927 21:07:57.968902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:07:57.969105 392 external_mini_cluster.cc:1428] Setting key 79c244f643d35fdc871cb5afe6226e80
I20260927 21:07:57.969167 10654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a: Bootstrap starting.
I20260927 21:07:57.969894 10654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:57.970208 10654 log.cc:824] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:57.970903 10654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a: No bootstrap required, opened a new log
I20260927 21:07:57.973147 10654 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5e8671169b49c9baebabb2f359be2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34351 } }
I20260927 21:07:57.973299 10654 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:57.973482 10654 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5e8671169b49c9baebabb2f359be2a, State: Initialized, Role: FOLLOWER
I20260927 21:07:57.973654 10654 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5e8671169b49c9baebabb2f359be2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34351 } }
I20260927 21:07:57.973778 10654 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:07:57.973830 10654 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:07:57.973877 10654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:57.974848 10654 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5e8671169b49c9baebabb2f359be2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34351 } }
I20260927 21:07:57.974937 10654 leader_election.cc:302] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5e8671169b49c9baebabb2f359be2a; no voters:
I20260927 21:07:57.975131 10654 leader_election.cc:288] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:07:57.975138 10659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:57.975266 10659 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [term 1 LEADER]: Becoming Leader. State: Replica: de5e8671169b49c9baebabb2f359be2a, State: Running, Role: LEADER
I20260927 21:07:57.975345 10659 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5e8671169b49c9baebabb2f359be2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34351 } }
I20260927 21:07:57.975397 10654 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:07:57.975666 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5e8671169b49c9baebabb2f359be2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5e8671169b49c9baebabb2f359be2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34351 } } }
I20260927 21:07:57.975824 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [sys.catalog]: This master's current role is: LEADER
I20260927 21:07:57.976024 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5e8671169b49c9baebabb2f359be2a. Latest consensus state: current_term: 1 leader_uuid: "de5e8671169b49c9baebabb2f359be2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5e8671169b49c9baebabb2f359be2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34351 } } }
I20260927 21:07:57.976099 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a [sys.catalog]: This master's current role is: LEADER
Sep 27 21:07:57 dist-test-slave-8bgs krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:57.976410 10666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:07:57.976938 10666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:07:57.978843 10666 catalog_manager.cc:1347] Generated new cluster ID: fcb0bcdf45e4482080a98e6808f85e47
I20260927 21:07:57.978897 10666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:07:57.980448 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:57.969569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.0.98.62:34351)
0927 21:07:57.969759 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:57.969767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:57.969799 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:07:57.970402 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:57.970411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:57.970477 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:57.970601 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:57.971280 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:57.972586 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:57.973314 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:57.973944 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:57.974058 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:57.976907 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:57.976911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:57.976911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:57.976929 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:57.978959 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:57.979328 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:57.979331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:57.979331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:57.979426 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:57.979643 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:57.979644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:57.979644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:57.979856 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:57.980020 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:57.980121 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:07:57.980246 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260927 21:07:57.993489 10666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:07:57.994164 10666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:07:57.999549 10666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a: Generated new TSK 0
I20260927 21:07:57.999717 10666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:07:58.030542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34351
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:58.134830 10680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:58.135064 10680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:58.135100 10680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:58.136243 10680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:58.136309 10680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:58.136354 10680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:07:58.137797 10680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:58.138051 10680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:58.138270 10680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:58.143476 10685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:58.143523 10686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:58.143648 10680 server_base.cc:1034] running on GCE node
W20260927 21:07:58.143522 10688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:58.143880 10680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:58.144129 10680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:58.145278 10680 hybrid_clock.cc:648] HybridClock initialized: now 1790543278145263 us; error 27 us; skew 500 ppm
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:58.147655 10680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:07:58.148056 10680 webserver.cc:460] Webserver started at http://127.0.98.1:44645/ using document root <none> and password file <none>
I20260927 21:07:58.148272 10680 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:58.148339 10680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:58.148489 10680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:58.149580 10680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff4f9ecfc08346b180b5867b9904c892"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "52383e43eee65f95aac630fe90210634"
server_key_iv: "3c51ff6dcdf11540a817ff1219c82194"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.149935 10680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff4f9ecfc08346b180b5867b9904c892"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "52383e43eee65f95aac630fe90210634"
server_key_iv: "3c51ff6dcdf11540a817ff1219c82194"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.151222 10680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:07:58.151933 10695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.152097 10680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:58.152171 10680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "ff4f9ecfc08346b180b5867b9904c892"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "52383e43eee65f95aac630fe90210634"
server_key_iv: "3c51ff6dcdf11540a817ff1219c82194"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.152220 10680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:58.178993 10680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:58.181382 10680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:58.181501 10680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:58.181677 10680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:58.181990 10680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:58.182030 10680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.182053 10680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:58.182101 10680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.187901 10680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45745
I20260927 21:07:58.187973 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45745 every 8 connection(s)
I20260927 21:07:58.188311 10680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:58.195928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10680
I20260927 21:07:58.196049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:07:58.196275 392 external_mini_cluster.cc:1428] Setting key 78121469c4cc75bf80ec1ad4ba0b2c1e
I20260927 21:07:58.197504 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.189053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45751 (local address 127.0.98.62:34351)
0927 21:07:58.189120 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.189123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.189321 (+ 198us) server_negotiation.cc:407] Connection header received
0927 21:07:58.189776 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.189777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.189786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.189839 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:58.190734 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.191253 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.192204 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.192394 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.192442 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.194445 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:58.194446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:58.194447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:58.194453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:58.195915 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.196449 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.196450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.196450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.196518 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.196796 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.196797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.196797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.196932 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:58.197005 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.197359 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.197441 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:07:58.197798 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.189027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34351 (local address 127.0.98.1:45751)
0927 21:07:58.189200 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.189236 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.189693 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.189895 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.189898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.190462 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:58.190655 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.190660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.191327 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.191327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.192122 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.192124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.192465 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.192466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.192566 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.193029 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:58.193037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:58.194366 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:58.195999 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:58.196006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:58.196009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:58.196311 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:58.196590 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:58.196591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:58.196592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:58.196719 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:58.197022 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:58.197029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:58.197199 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.197326 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.197513 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260927 21:07:58.198225 10809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34351
I20260927 21:07:58.198330 10809 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:58.198539 10809 heartbeater.cc:511] Master 127.0.98.62:34351 requested a full tablet report, sending...
I20260927 21:07:58.199055 10618 ts_manager.cc:194] Registered new tserver with Master: ff4f9ecfc08346b180b5867b9904c892 (127.0.98.1:45745)
I20260927 21:07:58.199741 10618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45751
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:07:58.218987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34351
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:58.299561 10814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:58.299763 10814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:58.299785 10814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:58.300933 10814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:58.300977 10814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:58.301031 10814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:07:58.302702 10814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:58.302968 10814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:58.303241 10814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:58.308090 10819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:58.308117 10814 server_base.cc:1034] running on GCE node
W20260927 21:07:58.308101 10820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:58.308112 10822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:58.308604 10814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:58.308821 10814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:58.309957 10814 hybrid_clock.cc:648] HybridClock initialized: now 1790543278309947 us; error 26 us; skew 500 ppm
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:58.312510 10814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:07:58.312913 10814 webserver.cc:460] Webserver started at http://127.0.98.2:40875/ using document root <none> and password file <none>
I20260927 21:07:58.313130 10814 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:58.313174 10814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:58.313321 10814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:58.314320 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "c47eeb78861b4147909ccee2709867dc"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "d3e9a2a7659b358fae0da45244e01f28"
server_key_iv: "7cd2e7a69010285c38c775a89a81c393"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.314674 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c47eeb78861b4147909ccee2709867dc"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "d3e9a2a7659b358fae0da45244e01f28"
server_key_iv: "7cd2e7a69010285c38c775a89a81c393"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.316012 10814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:07:58.316699 10829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.316872 10814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:07:58.316941 10814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "c47eeb78861b4147909ccee2709867dc"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "d3e9a2a7659b358fae0da45244e01f28"
server_key_iv: "7cd2e7a69010285c38c775a89a81c393"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.317045 10814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:58.325335 10814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:58.326965 10814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:58.327138 10814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:58.327344 10814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:58.327636 10814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:58.327669 10814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.327688 10814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:58.327734 10814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.333155 10814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44397
I20260927 21:07:58.333218 10942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44397 every 8 connection(s)
I20260927 21:07:58.333521 10814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:58.341919 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.334188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60971 (local address 127.0.98.62:34351)
0927 21:07:58.334271 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.334273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.334438 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:07:58.334938 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.334940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.334951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.335020 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:58.335950 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.336462 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.337427 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.337638 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.337724 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.339685 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:58.339686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:58.339687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:58.339695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:58.340866 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.341202 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.341202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.341203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.341249 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.341443 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.341444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.341444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.341539 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:58.341588 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.341798 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.341858 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:07:58.342056 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.334142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34351 (local address 127.0.98.2:60971)
0927 21:07:58.334311 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.334358 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.334835 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.335108 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.335112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.335672 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:58.335846 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.335850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.336559 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.336560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.337335 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.337337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.337732 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.337733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.337855 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.338241 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:58.338248 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:58.339599 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:58.340933 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:58.340939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:58.340940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:58.341139 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:58.341304 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:58.341305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:58.341305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:58.341391 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:58.341592 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:58.341595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:58.341697 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.341772 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.341880 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260927 21:07:58.342561 10943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34351
I20260927 21:07:58.342634 10943 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:58.342818 10943 heartbeater.cc:511] Master 127.0.98.62:34351 requested a full tablet report, sending...
I20260927 21:07:58.343011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10814
I20260927 21:07:58.343146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:07:58.343286 10618 ts_manager.cc:194] Registered new tserver with Master: c47eeb78861b4147909ccee2709867dc (127.0.98.2:44397)
I20260927 21:07:58.343371 392 external_mini_cluster.cc:1428] Setting key f9c3888d4fb11fa584278e786eca3502
I20260927 21:07:58.343863 10618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60971
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:07:58.365681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34351
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:07:58.444656 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:07:58.444886 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:07:58.444922 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:07:58.446038 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:07:58.446101 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:07:58.446144 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:07:58.447618 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:07:58.447862 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:07:58.448079 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:07:58.453060 10954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:58.453109 10956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:07:58.453058 10953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:07:58.453406 10948 server_base.cc:1034] running on GCE node
I20260927 21:07:58.453569 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:07:58.453756 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:07:58.454919 10948 hybrid_clock.cc:648] HybridClock initialized: now 1790543278454903 us; error 27 us; skew 500 ppm
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:07:58.464093 10948 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:07:58.464578 10948 webserver.cc:460] Webserver started at http://127.0.98.3:38877/ using document root <none> and password file <none>
I20260927 21:07:58.464799 10948 fs_manager.cc:359] Metadata directory not provided
I20260927 21:07:58.464844 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:07:58.465013 10948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:07:58.466038 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "9bbb976be09f4b7ab08ff9629845796d"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "a796a1e2f2d50e15a0771bcb29b5cb6f"
server_key_iv: "08f42ab6329fb62812daf5ef4a568a7f"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.466396 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9bbb976be09f4b7ab08ff9629845796d"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "a796a1e2f2d50e15a0771bcb29b5cb6f"
server_key_iv: "08f42ab6329fb62812daf5ef4a568a7f"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.467739 10948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:07:58.468433 10963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.468613 10948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:07:58.468701 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "9bbb976be09f4b7ab08ff9629845796d"
format_stamp: "Formatted at 2026-09-27 21:07:58 on dist-test-slave-8bgs"
server_key: "a796a1e2f2d50e15a0771bcb29b5cb6f"
server_key_iv: "08f42ab6329fb62812daf5ef4a568a7f"
server_key_version: "encryptionkey@0"
I20260927 21:07:58.468772 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:07:58.491392 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:07:58.493134 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:07:58.493288 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:07:58.493515 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:07:58.493819 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:07:58.493873 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.493916 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:07:58.493943 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:07:58.499782 10948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32871
I20260927 21:07:58.499831 11076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32871 every 8 connection(s)
I20260927 21:07:58.500180 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:07:58.508881 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.500863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38753 (local address 127.0.98.62:34351)
0927 21:07:58.500923 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.500925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.501146 (+ 221us) server_negotiation.cc:407] Connection header received
0927 21:07:58.501678 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.501679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.501693 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.501744 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:58.502679 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.503272 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.504258 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.504473 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.504574 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.506482 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:58.506483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:58.506483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:58.506490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:58.507812 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.508151 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.508152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.508152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.508202 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.508401 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.508401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.508402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.508497 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:58.508545 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.508762 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.508818 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:07:58.509013 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.500858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34351 (local address 127.0.98.3:38753)
0927 21:07:58.501030 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.501064 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.501566 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.501818 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.501822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.502378 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:07:58.502570 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.502575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.503363 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.503364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.504154 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.504156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.504583 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.504585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.504702 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.505110 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:07:58.505118 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:07:58.506391 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:07:58.507883 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:58.507888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:58.507889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:58.508088 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:58.508260 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:07:58.508261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:07:58.508261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:07:58.508348 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:07:58.508549 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:07:58.508552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:07:58.508654 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.508729 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.508836 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260927 21:07:58.509430 11077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34351
I20260927 21:07:58.509524 11077 heartbeater.cc:464] Registering TS with master...
I20260927 21:07:58.509656 11077 heartbeater.cc:511] Master 127.0.98.62:34351 requested a full tablet report, sending...
I20260927 21:07:58.510017 10617 ts_manager.cc:194] Registered new tserver with Master: 9bbb976be09f4b7ab08ff9629845796d (127.0.98.3:32871)
I20260927 21:07:58.510203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 10948
I20260927 21:07:58.510283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:07:58.510456 392 external_mini_cluster.cc:1428] Setting key 8dbc8bc8d8ff243f8a5d31e1039fe145
I20260927 21:07:58.510532 10617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38753
I20260927 21:07:58.511301 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:07:58.517814 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.512493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.0.98.62:34351)
0927 21:07:58.512546 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.512547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.512666 (+ 119us) server_negotiation.cc:407] Connection header received
0927 21:07:58.512704 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.512704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.512711 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.512764 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:07:58.513240 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.513678 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.514372 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.514547 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.514594 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.515662 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:07:58.515663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:07:58.515663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:07:58.515668 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:07:58.516984 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.517253 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.517254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.517254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.517298 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:07:58.517450 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:07:58.517451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:07:58.517451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:07:58.517560 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:07:58.517626 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.517712 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.517763 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:07:58.519393 10617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:07:58.520979 10617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:07:58.526913 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.523050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.0.98.1:45745)
0927 21:07:58.523280 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.523284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.523423 (+ 139us) server_negotiation.cc:407] Connection header received
0927 21:07:58.523533 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.523535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.523550 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.523646 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:58.523863 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.524621 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.525811 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.526501 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.526533 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.526567 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.526787 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.526832 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":145,"threads_started":1}
I20260927 21:07:58.526970 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.522944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32871 (local address 127.0.0.1:40302)
0927 21:07:58.523300 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.523328 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.523408 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.523672 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.523674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.523676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:58.523758 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.523759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.525035 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.525035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.525978 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.525980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.526741 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.526743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.526797 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.526800 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.526846 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.526897 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":71,"threads_started":1}
I20260927 21:07:58.527173 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.523010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.98.3:32871)
0927 21:07:58.523186 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.523190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.523393 (+ 203us) server_negotiation.cc:407] Connection header received
0927 21:07:58.523500 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.523502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.523518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.523593 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:58.523845 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.524921 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.526058 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.526652 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.526726 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.526773 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.526870 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.526908 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260927 21:07:58.527511 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.522942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45745 (local address 127.0.0.1:48400)
0927 21:07:58.523290 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.523328 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.523407 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.523698 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.523699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.523701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:58.523770 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.523772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.524710 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.524710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.525695 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.525697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.526612 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.526612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.526662 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.526670 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.527346 (+ 676us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.527411 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":51,"threads_started":1}
I20260927 21:07:58.527558 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.523383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.0.98.2:44397)
0927 21:07:58.523767 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.523771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.523781 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:07:58.523817 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.523819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.523835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.523906 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:58.524105 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.525112 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.526356 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.527075 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.527161 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.527216 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.527430 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.527475 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":82,"threads_started":1}
I20260927 21:07:58.527529 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.523162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44397 (local address 127.0.0.1:34394)
0927 21:07:58.523518 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.523527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.523565 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.523979 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.523980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.523981 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:58.524018 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.524019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.525209 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.525210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.526273 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.526275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.527255 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.527256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.527322 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.527326 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.527381 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.527425 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":80,"threads_started":1}
I20260927 21:07:58.527835 11011 tablet_service.cc:1467] Processing CreateTablet for tablet cbec26d185854404b44a3be1d757ce73 (DEFAULT_TABLE table=test-table [id=7b6bf948bbfb49a2b1887646f675f139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:58.528220 11011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbec26d185854404b44a3be1d757ce73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:58.528416 10743 tablet_service.cc:1467] Processing CreateTablet for tablet cbec26d185854404b44a3be1d757ce73 (DEFAULT_TABLE table=test-table [id=7b6bf948bbfb49a2b1887646f675f139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:58.528614 10877 tablet_service.cc:1467] Processing CreateTablet for tablet cbec26d185854404b44a3be1d757ce73 (DEFAULT_TABLE table=test-table [id=7b6bf948bbfb49a2b1887646f675f139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:07:58.528806 10743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbec26d185854404b44a3be1d757ce73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:58.528896 10877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbec26d185854404b44a3be1d757ce73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:07:58.530777 11094 tablet_bootstrap.cc:492] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Bootstrap starting.
I20260927 21:07:58.531507 11094 tablet_bootstrap.cc:654] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:58.531934 11095 tablet_bootstrap.cc:492] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892: Bootstrap starting.
I20260927 21:07:58.532106 11094 log.cc:824] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:58.532632 11095 tablet_bootstrap.cc:654] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:58.532861 11094 tablet_bootstrap.cc:492] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: No bootstrap required, opened a new log
I20260927 21:07:58.532939 11094 ts_tablet_manager.cc:1397] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:58.533190 11095 log.cc:824] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:58.533524 11096 tablet_bootstrap.cc:492] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Bootstrap starting.
I20260927 21:07:58.533859 11095 tablet_bootstrap.cc:492] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892: No bootstrap required, opened a new log
I20260927 21:07:58.533931 11095 ts_tablet_manager.cc:1397] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:58.534452 11096 tablet_bootstrap.cc:654] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Neither blocks nor log segments found. Creating new log.
I20260927 21:07:58.534991 11096 log.cc:824] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:07:58.535151 11094 raft_consensus.cc:348] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.535351 11094 raft_consensus.cc:374] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:58.535398 11094 raft_consensus.cc:729] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbb976be09f4b7ab08ff9629845796d, State: Initialized, Role: FOLLOWER
I20260927 21:07:58.535521 11094 consensus_queue.cc:260] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.535530 11095 raft_consensus.cc:348] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.535715 11095 raft_consensus.cc:374] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:58.535768 11096 tablet_bootstrap.cc:492] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: No bootstrap required, opened a new log
I20260927 21:07:58.535763 11095 raft_consensus.cc:729] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4f9ecfc08346b180b5867b9904c892, State: Initialized, Role: FOLLOWER
I20260927 21:07:58.535790 11094 ts_tablet_manager.cc:1428] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:07:58.535835 11096 ts_tablet_manager.cc:1397] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:58.535902 11077 heartbeater.cc:503] Master 127.0.98.62:34351 was elected leader, sending a full tablet report...
I20260927 21:07:58.535907 11095 consensus_queue.cc:260] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.536166 11095 ts_tablet_manager.cc:1428] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:07:58.536206 10809 heartbeater.cc:503] Master 127.0.98.62:34351 was elected leader, sending a full tablet report...
I20260927 21:07:58.537771 11096 raft_consensus.cc:348] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.537894 11096 raft_consensus.cc:374] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:07:58.537914 11096 raft_consensus.cc:729] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47eeb78861b4147909ccee2709867dc, State: Initialized, Role: FOLLOWER
I20260927 21:07:58.538028 11096 consensus_queue.cc:260] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.538283 11096 ts_tablet_manager.cc:1428] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:07:58.538300 10943 heartbeater.cc:503] Master 127.0.98.62:34351 was elected leader, sending a full tablet report...
W20260927 21:07:58.584079 10944 tablet.cc:2369] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:07:58.594408 11102 raft_consensus.cc:482] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:07:58.594583 11102 raft_consensus.cc:504] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.595036 11102 leader_election.cc:288] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff4f9ecfc08346b180b5867b9904c892 (127.0.98.1:45745), 9bbb976be09f4b7ab08ff9629845796d (127.0.98.3:32871)
I20260927 21:07:58.598680 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.595287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57459 (local address 127.0.98.3:32871)
0927 21:07:58.595331 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.595333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.595494 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:07:58.595554 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.595555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.595567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.595623 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:58.595787 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.596473 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.597929 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.598347 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.598429 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.598440 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.598579 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.598612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:07:58.598996 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.595192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45745 (local address 127.0.98.2:45677)
0927 21:07:58.595247 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.595259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.595301 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.595566 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.595567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.595569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:58.595639 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.595641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.596570 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.596571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.597929 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.597931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.598737 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.598738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.598820 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.598823 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.598876 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.598909 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":31}
I20260927 21:07:58.598719 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.595192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32871 (local address 127.0.98.2:57459)
0927 21:07:58.595440 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.595450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:07:58.595501 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:07:58.595627 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:07:58.595627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:07:58.595628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:07:58.595697 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.595698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.596558 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.596559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.597844 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:07:58.597847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.598428 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.598429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:07:58.598507 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.598511 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:07:58.598566 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:07:58.598609 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":86,"thread_start_us":76,"threads_started":1}
I20260927 21:07:58.599123 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.595338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45677 (local address 127.0.98.1:45745)
0927 21:07:58.595385 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.595387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.595395 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:07:58.595424 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.595425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.595438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.595509 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:07:58.595723 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.596474 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.598024 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.598652 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.598739 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.598750 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.598960 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.599008 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:07:58.599514 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec26d185854404b44a3be1d757ce73" candidate_uuid: "c47eeb78861b4147909ccee2709867dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4f9ecfc08346b180b5867b9904c892" is_pre_election: true
I20260927 21:07:58.599675 10763 raft_consensus.cc:2454] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c47eeb78861b4147909ccee2709867dc in term 0.
I20260927 21:07:58.599882 10833 leader_election.cc:302] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c47eeb78861b4147909ccee2709867dc, ff4f9ecfc08346b180b5867b9904c892; no voters:
I20260927 21:07:58.600009 11102 raft_consensus.cc:2781] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:07:58.600090 11102 raft_consensus.cc:482] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:07:58.600112 11102 raft_consensus.cc:3037] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:58.600482 11031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec26d185854404b44a3be1d757ce73" candidate_uuid: "c47eeb78861b4147909ccee2709867dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbb976be09f4b7ab08ff9629845796d" is_pre_election: true
I20260927 21:07:58.600642 11031 raft_consensus.cc:2454] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c47eeb78861b4147909ccee2709867dc in term 0.
I20260927 21:07:58.601094 11102 raft_consensus.cc:504] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.601250 11102 leader_election.cc:288] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [CANDIDATE]: Term 1 election: Requested vote from peers ff4f9ecfc08346b180b5867b9904c892 (127.0.98.1:45745), 9bbb976be09f4b7ab08ff9629845796d (127.0.98.3:32871)
I20260927 21:07:58.601378 11031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec26d185854404b44a3be1d757ce73" candidate_uuid: "c47eeb78861b4147909ccee2709867dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbb976be09f4b7ab08ff9629845796d"
I20260927 21:07:58.601384 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec26d185854404b44a3be1d757ce73" candidate_uuid: "c47eeb78861b4147909ccee2709867dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4f9ecfc08346b180b5867b9904c892"
I20260927 21:07:58.601459 10763 raft_consensus.cc:3037] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:58.601460 11031 raft_consensus.cc:3037] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:07:58.602092 10763 raft_consensus.cc:2454] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c47eeb78861b4147909ccee2709867dc in term 1.
I20260927 21:07:58.602265 10833 leader_election.cc:302] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c47eeb78861b4147909ccee2709867dc, ff4f9ecfc08346b180b5867b9904c892; no voters:
I20260927 21:07:58.602346 11102 raft_consensus.cc:2781] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:07:58.602464 11102 raft_consensus.cc:686] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 1 LEADER]: Becoming Leader. State: Replica: c47eeb78861b4147909ccee2709867dc, State: Running, Role: LEADER
I20260927 21:07:58.602448 11031 raft_consensus.cc:2454] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c47eeb78861b4147909ccee2709867dc in term 1.
I20260927 21:07:58.602643 11102 consensus_queue.cc:237] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } }
I20260927 21:07:58.603492 10617 catalog_manager.cc:5495] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc reported cstate change: term changed from 0 to 1, leader changed from <none> to c47eeb78861b4147909ccee2709867dc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c47eeb78861b4147909ccee2709867dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c47eeb78861b4147909ccee2709867dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44397 } health_report { overall_health: HEALTHY } } }
I20260927 21:07:58.623473 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.620956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.0.98.2:44397)
0927 21:07:58.620999 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:07:58.621001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:07:58.621010 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:07:58.621082 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:07:58.621083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:07:58.621103 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:07:58.621201 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:07:58.621342 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.621809 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.622595 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:07:58.622803 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:07:58.622889 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:07:58.623071 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:07:58.623202 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:07:58.623306 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:07:58.623346 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 21:07:58.623368 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:07:58.625125 10763 raft_consensus.cc:1261] T cbec26d185854404b44a3be1d757ce73 P ff4f9ecfc08346b180b5867b9904c892 [term 1 FOLLOWER]: Refusing update from remote peer c47eeb78861b4147909ccee2709867dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:58.625432 11031 raft_consensus.cc:1261] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 1 FOLLOWER]: Refusing update from remote peer c47eeb78861b4147909ccee2709867dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:07:58.625485 11102 consensus_queue.cc:1035] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4f9ecfc08346b180b5867b9904c892" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:58.625754 11102 consensus_queue.cc:1035] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbb976be09f4b7ab08ff9629845796d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:07:58.627736 11107 mvcc.cc:205] Tried to move back new op lower bound from 7334065269245947904 to 7334065269157064704. Current Snapshot: MvccSnapshot[applied={T|T < 7334065269245947904}]
I20260927 21:07:58.628890 11110 mvcc.cc:205] Tried to move back new op lower bound from 7334065269245947904 to 7334065269157064704. Current Snapshot: MvccSnapshot[applied={T|T < 7334065269245947904 or (T in {7334065269245947904})}]
I20260927 21:07:58.629838 10617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:58.629932 10617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:07:58.630627 10617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5e8671169b49c9baebabb2f359be2a: Sending DeleteTablet for 3 replicas of tablet cbec26d185854404b44a3be1d757ce73
I20260927 21:07:58.630927 10743 tablet_service.cc:1514] Processing DeleteTablet for tablet cbec26d185854404b44a3be1d757ce73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48400
I20260927 21:07:58.630966 10877 tablet_service.cc:1514] Processing DeleteTablet for tablet cbec26d185854404b44a3be1d757ce73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34394
I20260927 21:07:58.630971 11011 tablet_service.cc:1514] Processing DeleteTablet for tablet cbec26d185854404b44a3be1d757ce73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:07:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40302
I20260927 21:07:58.631222 11118 tablet_replica.cc:330] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: stopping tablet replica
I20260927 21:07:58.631323 11116 tablet_replica.cc:330] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: stopping tablet replica
I20260927 21:07:58.631322 11118 raft_consensus.cc:2229] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:07:58.631397 11116 raft_consensus.cc:2229] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:07:58.631421 11118 raft_consensus.cc:2258] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:58.631485 11116 raft_consensus.cc:2258] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:07:58.631538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10680
I20260927 21:07:58.632514 11118 ts_tablet_manager.cc:1905] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:58.632576 11116 ts_tablet_manager.cc:1905] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:07:58.634409 11118 ts_tablet_manager.cc:1918] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:58.634409 11116 ts_tablet_manager.cc:1918] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:07:58.634473 11118 log.cc:1196] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/cbec26d185854404b44a3be1d757ce73
I20260927 21:07:58.634478 11116 log.cc:1196] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/cbec26d185854404b44a3be1d757ce73
I20260927 21:07:58.634752 11118 ts_tablet_manager.cc:1939] T cbec26d185854404b44a3be1d757ce73 P 9bbb976be09f4b7ab08ff9629845796d: Deleting consensus metadata
I20260927 21:07:58.634752 11116 ts_tablet_manager.cc:1939] T cbec26d185854404b44a3be1d757ce73 P c47eeb78861b4147909ccee2709867dc: Deleting consensus metadata
I20260927 21:07:58.635221 10605 catalog_manager.cc:4841] TS c47eeb78861b4147909ccee2709867dc (127.0.98.2:44397): tablet cbec26d185854404b44a3be1d757ce73 (table test-table [id=7b6bf948bbfb49a2b1887646f675f139]) successfully deleted
I20260927 21:07:58.635252 10603 catalog_manager.cc:4841] TS 9bbb976be09f4b7ab08ff9629845796d (127.0.98.3:32871): tablet cbec26d185854404b44a3be1d757ce73 (table test-table [id=7b6bf948bbfb49a2b1887646f675f139]) successfully deleted
W20260927 21:07:58.639488 10605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45745 (error 108)
I20260927 21:07:58.639714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10814
I20260927 21:07:58.640421 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:07:58.639859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45745 (local address 127.0.0.1:48404)
0927 21:07:58.640354 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 21:07:58.640388 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
W20260927 21:07:58.641386 10605 catalog_manager.cc:4584] TS ff4f9ecfc08346b180b5867b9904c892 (127.0.98.1:45745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbec26d185854404b44a3be1d757ce73: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45745: connect: Connection refused (error 111)
I20260927 21:07:58.644248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10948
I20260927 21:07:58.648591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 10586
2026-09-27T21:07:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1922 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[11122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[11122](info): set up 2 sockets
Sep 27 21:07:58 dist-test-slave-8bgs krb5kdc[11122](info): commencing operation
krb5kdc: starting...
W20260927 21:07:59.676172 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:07:59 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:07:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 21:07:59.829483 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6L0Aex/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/postgres -l logfile start
2026-09-27 21:08:02.103 UTC [11152] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:08:02.103 UTC [11152] LOG: listening on IPv4 address "127.0.98.20", port 42131
2026-09-27 21:08:02.106 UTC [11152] LOG: listening on Unix socket "/tmp/.s.PGSQL.42131"
2026-09-27 21:08:02.140 UTC [11155] LOG: database system was shut down at 2026-09-27 21:08:00 UTC
2026-09-27 21:08:02.144 UTC [11152] LOG: database system is ready to accept connections
I20260927 21:08:03.087812 392 mini_postgres.cc:96] Postgres bound to 42131
2026-09-27 21:08:03.093 UTC [11163] FATAL: database "slave" does not exist
127.0.98.20:42131 - accepting connections
I20260927 21:08:03.094092 392 mini_ranger.cc:161] Starting Ranger...
I20260927 21:08:03.107900 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:08:03.508287 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:08:03.508371 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-admin
2026-09-27 21:08:03,784 [I] DB FLAVOR :POSTGRES
2026-09-27 21:08:03,785 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:08:03,785 [I] Checking connection..
2026-09-27 21:08:03,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:08:04,038 [I] Checking connection passed.
2026-09-27 21:08:04,039 [I] --------- Verifying version history table ---------
2026-09-27 21:08:04,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:08:04,297 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:08:04,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:08:04,550 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:08:04,551 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:08:04,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:08:04.798 UTC [11227] WARNING: there is no transaction in progress
2026-09-27 21:08:04,805 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:08:04,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:08:05,043 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:08:05,043 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:08:05,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:08:05,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:08:05,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8bgs.c.gcp-upstream.internal','N') ;"
2026-09-27 21:08:05,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:08:06,000 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:08:06,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:08:06,248 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:08:06,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:08:06,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:08:06,726 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:08:06,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:08:07.619 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.634 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.650 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.665 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.679 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.698 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.755 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.762 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.776 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.787 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.796 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.809 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.818 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.824 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.830 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:07.837 UTC [11335] WARNING: there is no transaction in progress
2026-09-27 21:08:08,416 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:08:08,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:08:08,639 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:08:08,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:08:08,905 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:08:08,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:08:09,159 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:08:09,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:08:09,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:08:09,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8bgs.c.gcp-upstream.internal';"
2026-09-27 21:08:09,909 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:08:09,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:08:10,131 [I] DB_PATCHES have already been applied
I20260927 21:08:10.137269 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6L0Aex/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:08:10.137327 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 21:08:10.140290 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:49795
Sep 27, 2026 9:08:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:08:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:08:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:08:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:08:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:08:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:08:11.264048 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.170s user 0.001s sys 0.007s
Sep 27, 2026 9:08:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:08:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 9:08:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:08:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:08:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:08:33.038141 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:08:33 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543313, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:08:33.118162 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:08:33.127490 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:08:33.938302 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:08:33.938371 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms
I20260927 21:08:33.938385 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42131
2026-09-27 21:08:34,109 [I] DB FLAVOR :POSTGRES
2026-09-27 21:08:34,110 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:08:34,110 [I] Checking connection
2026-09-27 21:08:34,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:08:34,346 [I] connection success
2026-09-27 21:08:34,346 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:08:34,346 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:08:34,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:08:34,595 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:08:34,595 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:08:34,596 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:08:34,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42131/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:08:34,851 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:08:35.340811 392 mini_ranger_kms.cc:310] Created kms service
I20260927 21:08:35.529605 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:08:35.570248 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:08:35 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543313, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:08:35.957639 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:08:35.957826 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:08:35.957850 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 21:08:35.961468 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11515
I20260927 21:08:35.961568 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55001
Sep 27, 2026 9:08:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:08:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:08:38.185110 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.067s user 0.000s sys 0.154s
I20260927 21:08:38.185216 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:08:38.185315 392 mini_ranger_kms.cc:401] 127.0.98.20:55001/kms/v1/keys
Sep 27 21:08:38 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:08:38 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:08:39 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:08:41 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:08:41.357323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38011
--ranger_config_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55001/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:08:41.471306 11566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:08:41.471590 11566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:08:41.471622 11566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:08:41.473281 11566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:08:41.473336 11566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:08:41.473356 11566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:08:41.473374 11566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:08:41.473392 11566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:08:41.475567 11566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38011
--ranger_config_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:08:41.475798 11566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:08:41.476046 11566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:08:41.480804 11571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:41.481189 11566 server_base.cc:1034] running on GCE node
W20260927 21:08:41.480870 11574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:08:41.480886 11572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:41.481540 11566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:08:41.481828 11566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:08:41.482981 11566 hybrid_clock.cc:648] HybridClock initialized: now 1790543321482964 us; error 36 us; skew 500 ppm
Sep 27 21:08:41 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:08:41.485802 11566 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:08:41.486227 11566 webserver.cc:460] Webserver started at http://127.0.98.62:43345/ using document root <none> and password file <none>
I20260927 21:08:41.486450 11566 fs_manager.cc:359] Metadata directory not provided
I20260927 21:08:41.486509 11566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:08:41.486644 11566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:08:41 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:08:41.551054 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "dc31f098649f43189e5000cb643638c9"
format_stamp: "Formatted at 2026-09-27 21:08:41 on dist-test-slave-8bgs"
server_key: "ffb0d007bd87e154e01132a6e0b81fb5"
server_key_iv: "0336e6af06cf15adbd830f7f99d28864"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:41.551558 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc31f098649f43189e5000cb643638c9"
format_stamp: "Formatted at 2026-09-27 21:08:41 on dist-test-slave-8bgs"
server_key: "ffb0d007bd87e154e01132a6e0b81fb5"
server_key_iv: "0336e6af06cf15adbd830f7f99d28864"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:41.553089 11566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:08:41.564960 11581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:41.565137 11566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:08:41.565224 11566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "dc31f098649f43189e5000cb643638c9"
format_stamp: "Formatted at 2026-09-27 21:08:41 on dist-test-slave-8bgs"
server_key: "ffb0d007bd87e154e01132a6e0b81fb5"
server_key_iv: "0336e6af06cf15adbd830f7f99d28864"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:41.565337 11566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:08:41.581910 11566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:08:41.583998 11566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:08:41.584153 11566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:08:41.589370 11566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38011
I20260927 21:08:41.589391 11633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38011 every 8 connection(s)
I20260927 21:08:41.589752 11566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:08:41.590407 11634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:08:41.592703 11634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9: Bootstrap starting.
I20260927 21:08:41.593305 11634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9: Neither blocks nor log segments found. Creating new log.
I20260927 21:08:41.593564 11634 log.cc:824] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:08:41.594260 11634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9: No bootstrap required, opened a new log
I20260927 21:08:41.596312 11634 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc31f098649f43189e5000cb643638c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260927 21:08:41.596446 11634 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:08:41.596467 11634 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc31f098649f43189e5000cb643638c9, State: Initialized, Role: FOLLOWER
I20260927 21:08:41.596684 11634 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc31f098649f43189e5000cb643638c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260927 21:08:41.596725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 11566
I20260927 21:08:41.596768 11634 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:08:41.596803 11634 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:08:41.596819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:08:41.596866 11634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:08:41.596988 392 external_mini_cluster.cc:1428] Setting key d59afa2d97adcb7eca3b188cca92359f
I20260927 21:08:41.597615 11634 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc31f098649f43189e5000cb643638c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260927 21:08:41.597790 11634 leader_election.cc:302] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc31f098649f43189e5000cb643638c9; no voters:
I20260927 21:08:41.597967 11634 leader_election.cc:288] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:08:41.598254 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc31f098649f43189e5000cb643638c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } } }
I20260927 21:08:41.598304 11634 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:08:41.598531 11639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:08:41.598692 11639 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [term 1 LEADER]: Becoming Leader. State: Replica: dc31f098649f43189e5000cb643638c9, State: Running, Role: LEADER
I20260927 21:08:41.598390 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:08:41.600431 11639 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc31f098649f43189e5000cb643638c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } }
I20260927 21:08:41.601289 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc31f098649f43189e5000cb643638c9. Latest consensus state: current_term: 1 leader_uuid: "dc31f098649f43189e5000cb643638c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc31f098649f43189e5000cb643638c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38011 } } }
I20260927 21:08:41.601378 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:08:41.601975 11634 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:08:41.602108 11653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 21:08:41 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:08:41.605073 11653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:08:41.610217 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:41.597465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.98.62:38011)
0927 21:08:41.597627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:41.597636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:41.597668 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:08:41.597828 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:41.597834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:41.597891 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:41.598031 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:08:41.598812 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:41.600265 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:41.601112 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:41.601802 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:41.601903 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:41.604800 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:08:41.604827 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:08:41.604827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:08:41.604859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:08:41.607948 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:41.608378 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:41.608381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:41.608382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:41.608485 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:41.608682 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:41.608683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:41.608683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:41.609045 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:08:41.609233 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:41.609768 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 21:08:41.609929 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260927 21:08:41.613899 11653 catalog_manager.cc:1347] Generated new cluster ID: 53fba9b1585a41689a1e173b5abf0d2a
I20260927 21:08:41.613969 11653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:08:41.624961 11653 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:08:41.625707 11653 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:08:41.630527 11653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9: Generated new TSK 0
I20260927 21:08:41.630774 11653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:08:42.370863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38011
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:08:42.487522 11679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:08:42.487712 11679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:08:42.487735 11679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:08:42.488860 11679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:08:42.488902 11679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:08:42.488929 11679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:08:42.490370 11679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:08:42.490588 11679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:08:42.490815 11679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:08:42.495586 11685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:08:42.495603 11687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:42.495746 11679 server_base.cc:1034] running on GCE node
W20260927 21:08:42.495895 11684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:42.496088 11679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:08:42.496366 11679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:08:42.497542 11679 hybrid_clock.cc:648] HybridClock initialized: now 1790543322497528 us; error 27 us; skew 500 ppm
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:08:42.500592 11679 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:08:42.501060 11679 webserver.cc:460] Webserver started at http://127.0.98.1:42783/ using document root <none> and password file <none>
I20260927 21:08:42.501292 11679 fs_manager.cc:359] Metadata directory not provided
I20260927 21:08:42.501355 11679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:08:42.501487 11679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:08:42.513670 11679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "af0dba6df1a64ffbad0b351339c8b6da"
format_stamp: "Formatted at 2026-09-27 21:08:42 on dist-test-slave-8bgs"
server_key: "be7eb0c76bcca6ef43310daf8f305a4a"
server_key_iv: "1a2c7b0baa5b99371509552ce71138d4"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:42.514151 11679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "af0dba6df1a64ffbad0b351339c8b6da"
format_stamp: "Formatted at 2026-09-27 21:08:42 on dist-test-slave-8bgs"
server_key: "be7eb0c76bcca6ef43310daf8f305a4a"
server_key_iv: "1a2c7b0baa5b99371509552ce71138d4"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:42.516001 11679 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:08:42.531221 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:42.531404 11679 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260927 21:08:42.531517 11679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "af0dba6df1a64ffbad0b351339c8b6da"
format_stamp: "Formatted at 2026-09-27 21:08:42 on dist-test-slave-8bgs"
server_key: "be7eb0c76bcca6ef43310daf8f305a4a"
server_key_iv: "1a2c7b0baa5b99371509552ce71138d4"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:42.531610 11679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:08:42.546902 11679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:08:42.548920 11679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:08:42.549088 11679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:08:42.549324 11679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:08:42.549687 11679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:08:42.549748 11679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:42.549793 11679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:08:42.549824 11679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:42.556314 11679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39611
I20260927 21:08:42.556771 11679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:08:42.557106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 11679
I20260927 21:08:42.557214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:08:42.557390 392 external_mini_cluster.cc:1428] Setting key 94549aed41e68cc5691b2785a51a7060
I20260927 21:08:42.566247 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39611 every 8 connection(s)
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:08:42.581782 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:42.566959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38011 (local address 127.0.98.1:37187)
0927 21:08:42.567132 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:42.567173 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:42.567837 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:42.571463 (+ 3626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:42.571468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:42.571712 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:08:42.571997 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:42.572003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.573053 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.573054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:42.574339 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:42.574341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.574925 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.574926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:42.575075 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:42.575729 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:08:42.575739 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:08:42.577448 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:08:42.579614 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:08:42.579620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:08:42.579622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:08:42.579916 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:08:42.580158 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:08:42.580159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:08:42.580159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:08:42.580299 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:08:42.580624 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:08:42.580632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:08:42.580824 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 21:08:42.580964 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 21:08:42.581532 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260927 21:08:42.582130 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:42.566976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37187 (local address 127.0.98.62:38011)
0927 21:08:42.571197 (+ 4221us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:42.571199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:42.571209 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:08:42.571237 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:42.571238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:42.571251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:42.571324 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:08:42.572108 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.572944 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.574433 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.574761 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.574859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:42.577559 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:08:42.577561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:08:42.577562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:08:42.577569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:08:42.579523 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:42.580019 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:42.580020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:42.580021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:42.580087 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:42.580380 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:42.580381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:42.580382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:42.580524 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:08:42.580600 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:42.581946 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0927 21:08:42.582033 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4185,"thread_start_us":88,"threads_started":1}
I20260927 21:08:42.582734 11810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38011
I20260927 21:08:42.582839 11810 heartbeater.cc:464] Registering TS with master...
I20260927 21:08:42.583039 11810 heartbeater.cc:511] Master 127.0.98.62:38011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 21:08:42.586244 11598 ts_manager.cc:194] Registered new tserver with Master: af0dba6df1a64ffbad0b351339c8b6da (127.0.98.1:39611)
I20260927 21:08:42.587204 11598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:08:42.599702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38011
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:08:42.724177 11816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:08:42.724588 11816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:08:42.724681 11816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:08:42.726583 11816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:08:42.726699 11816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:08:42.726790 11816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:08:42.729032 11816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:08:42.729486 11816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:08:42.729796 11816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:08:42.736483 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:08:42.736521 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:08:42.736536 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:42.737372 11816 server_base.cc:1034] running on GCE node
I20260927 21:08:42.737577 11816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:08:42.737829 11816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:08:42.739006 11816 hybrid_clock.cc:648] HybridClock initialized: now 1790543322738983 us; error 32 us; skew 500 ppm
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:08:42.742405 11816 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:08:42.742972 11816 webserver.cc:460] Webserver started at http://127.0.98.2:41805/ using document root <none> and password file <none>
I20260927 21:08:42.743258 11816 fs_manager.cc:359] Metadata directory not provided
I20260927 21:08:42.743315 11816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:08:42.743502 11816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:08:42.760751 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d5b20c21f78449a83865f257eddb639"
format_stamp: "Formatted at 2026-09-27 21:08:42 on dist-test-slave-8bgs"
server_key: "907d485b2cd40fc0ca981d718e706e28"
server_key_iv: "74bd5315765c6fb4939605d3167b02af"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:42.761215 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d5b20c21f78449a83865f257eddb639"
format_stamp: "Formatted at 2026-09-27 21:08:42 on dist-test-slave-8bgs"
server_key: "907d485b2cd40fc0ca981d718e706e28"
server_key_iv: "74bd5315765c6fb4939605d3167b02af"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:42.763055 11816 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:08:42.774595 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:42.774818 11816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:08:42.774955 11816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "5d5b20c21f78449a83865f257eddb639"
format_stamp: "Formatted at 2026-09-27 21:08:42 on dist-test-slave-8bgs"
server_key: "907d485b2cd40fc0ca981d718e706e28"
server_key_iv: "74bd5315765c6fb4939605d3167b02af"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:42.775190 11816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:08:42.801087 11816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:08:42.803606 11816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:08:42.803797 11816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:08:42.804059 11816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:08:42.804466 11816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:08:42.804550 11816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:42.804620 11816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:08:42.804674 11816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:42.813561 11816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36991
I20260927 21:08:42.814074 11816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:08:42.814468 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36991 every 8 connection(s)
I20260927 21:08:42.817194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 11816
I20260927 21:08:42.817299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:08:42.817487 392 external_mini_cluster.cc:1428] Setting key ba57627106fe25eae0b2375ba45a4402
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:08:42.832279 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:42.815187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42125 (local address 127.0.98.62:38011)
0927 21:08:42.815235 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:42.815237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:42.815955 (+ 718us) server_negotiation.cc:407] Connection header received
0927 21:08:42.816654 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:42.816655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:42.816666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:42.816696 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:08:42.817740 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.818552 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.820068 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.820382 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.820430 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:42.823804 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:08:42.823806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:08:42.823806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:08:42.823815 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:08:42.826034 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:42.826535 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:42.826536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:42.826537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:42.826600 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:42.831448 (+ 4848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:42.831450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:42.831450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:42.831600 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:08:42.831656 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:42.832079 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 21:08:42.832180 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 21:08:42.834946 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:42.815037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38011 (local address 127.0.98.2:42125)
0927 21:08:42.815818 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:42.815859 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:42.816489 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:42.816965 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:42.816969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:42.817205 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:08:42.817614 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:42.817639 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.818669 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.818670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:42.819968 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:42.819971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:42.820527 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:42.820528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:42.820662 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:42.821997 (+ 1335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:08:42.822007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:08:42.823711 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:08:42.826142 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:08:42.826148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:08:42.826151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:08:42.826451 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:08:42.831173 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:08:42.831174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:08:42.831174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:08:42.831350 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:08:42.831689 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:08:42.831695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:08:42.831884 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 21:08:42.834471 (+ 2587us) client_negotiation.cc:240] Negotiation successful
0927 21:08:42.834668 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":267,"threads_started":1}
I20260927 21:08:42.835723 11948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38011
I20260927 21:08:42.835824 11948 heartbeater.cc:464] Registering TS with master...
I20260927 21:08:42.836042 11948 heartbeater.cc:511] Master 127.0.98.62:38011 requested a full tablet report, sending...
I20260927 21:08:42.836448 11598 ts_manager.cc:194] Registered new tserver with Master: 5d5b20c21f78449a83865f257eddb639 (127.0.98.2:36991)
I20260927 21:08:42.836947 11598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:08:42.854678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38011
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:08:42.969584 11953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:08:42.969936 11953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:08:42.970037 11953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:08:42.971774 11953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:08:42.971837 11953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:08:42.971875 11953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:08:42.973943 11953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:08:42.974334 11953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:08:42.974648 11953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:08:42.980410 11958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:08:42.980785 11961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:42.984189 11953 server_base.cc:1034] running on GCE node
W20260927 21:08:42.987767 11959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:08:42.988080 11953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:08:42.988363 11953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:08:42.995739 11953 hybrid_clock.cc:648] HybridClock initialized: now 1790543322995716 us; error 32 us; skew 500 ppm
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:08:42.998973 11953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:08:42.999528 11953 webserver.cc:460] Webserver started at http://127.0.98.3:33081/ using document root <none> and password file <none>
I20260927 21:08:42.999806 11953 fs_manager.cc:359] Metadata directory not provided
I20260927 21:08:42.999892 11953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:08:43.000036 11953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:08:42 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:08:43.012439 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "908fe9a180b0441398c4d908911f3cf3"
format_stamp: "Formatted at 2026-09-27 21:08:43 on dist-test-slave-8bgs"
server_key: "a0add180d360963808ec082f32960000"
server_key_iv: "e9d26b26b40063ded869c335c9884dc3"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:43.012948 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "908fe9a180b0441398c4d908911f3cf3"
format_stamp: "Formatted at 2026-09-27 21:08:43 on dist-test-slave-8bgs"
server_key: "a0add180d360963808ec082f32960000"
server_key_iv: "e9d26b26b40063ded869c335c9884dc3"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:43.014364 11953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:08:43.023197 11968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:43.023401 11953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:08:43.023481 11953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "908fe9a180b0441398c4d908911f3cf3"
format_stamp: "Formatted at 2026-09-27 21:08:43 on dist-test-slave-8bgs"
server_key: "a0add180d360963808ec082f32960000"
server_key_iv: "e9d26b26b40063ded869c335c9884dc3"
server_key_version: "kuduclusterkey@0"
I20260927 21:08:43.023555 11953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:08:43.051877 11953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:08:43.054379 11953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:08:43.054572 11953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:08:43.054814 11953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:08:43.055225 11953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:08:43.055310 11953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:43.055370 11953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:08:43.055414 11953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:08:43.061836 11953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40975
I20260927 21:08:43.062323 11953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:08:43.065924 12081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40975 every 8 connection(s)
I20260927 21:08:43.070101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 11953
I20260927 21:08:43.070195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:08:43.070375 392 external_mini_cluster.cc:1428] Setting key 8a87fbaaf94abc1222c6220518bc2a2a
Sep 27 21:08:43 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:08:43.078333 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.066648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48689 (local address 127.0.98.62:38011)
0927 21:08:43.066716 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.066718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.067409 (+ 691us) server_negotiation.cc:407] Connection header received
0927 21:08:43.068115 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.068117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.068129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.068160 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:08:43.069029 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.069855 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.072114 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.072422 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.072463 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.074977 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:08:43.074979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:08:43.074980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:08:43.074987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:08:43.076742 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:43.077237 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:43.077239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:43.077239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:43.077296 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:43.077581 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:43.077582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:43.077582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:43.077712 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:08:43.077756 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.078153 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.078238 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:08:43.079123 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.066628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38011 (local address 127.0.98.3:48689)
0927 21:08:43.067276 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.067319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:43.067979 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:43.068347 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:43.068351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:43.068631 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:08:43.068907 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.068913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.070606 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.070607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.072012 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.072014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.072549 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.072550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.072685 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.073293 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:08:43.073303 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:08:43.074876 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:08:43.076837 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:08:43.076843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:08:43.076845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:08:43.077142 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:08:43.077370 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:08:43.077370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:08:43.077371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:08:43.077499 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:08:43.077802 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:08:43.077808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:08:43.077986 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 21:08:43.078675 (+ 689us) client_negotiation.cc:240] Negotiation successful
0927 21:08:43.078872 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":86,"threads_started":1}
I20260927 21:08:43.079799 12082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38011
I20260927 21:08:43.079907 12082 heartbeater.cc:464] Registering TS with master...
I20260927 21:08:43.080096 12082 heartbeater.cc:511] Master 127.0.98.62:38011 requested a full tablet report, sending...
I20260927 21:08:43.080461 11598 ts_manager.cc:194] Registered new tserver with Master: 908fe9a180b0441398c4d908911f3cf3 (127.0.98.3:40975)
I20260927 21:08:43.080983 11598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48689
I20260927 21:08:43.082432 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:08:43.099220 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.087418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.98.62:38011)
0927 21:08:43.087458 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.087459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.091275 (+ 3816us) server_negotiation.cc:407] Connection header received
0927 21:08:43.091380 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.091381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.091392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.091420 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:08:43.092250 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.093036 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.094164 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.094452 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.094494 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.096088 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:08:43.096089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:08:43.096089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:08:43.096096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:08:43.097787 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:43.098257 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:43.098258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:43.098259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:43.098316 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:08:43.098534 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:08:43.098535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:08:43.098535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:08:43.098664 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:08:43.098739 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.099011 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.099123 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:08:43.101643 11598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:08:43.104067 11598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:08:43.116986 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.107547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40975 (local address 127.0.0.1:37660)
0927 21:08:43.108699 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.108741 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:43.108803 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:43.109281 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:43.109284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:43.109286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:08:43.109372 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.109374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.113392 (+ 4018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.113393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.114538 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.114541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.116674 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.116675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.116758 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.116771 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:08:43.116838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:08:43.116886 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":71,"threads_started":1}
I20260927 21:08:43.119288 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.107749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39611 (local address 127.0.0.1:40452)
0927 21:08:43.110148 (+ 2399us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.110160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:43.110217 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:43.110748 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:43.110748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:43.110750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:08:43.110805 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.110806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.112118 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.112119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.117511 (+ 5392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.117514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.118583 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.118584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.119056 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.119061 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:08:43.119174 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:08:43.119214 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2382,"thread_start_us":57,"threads_started":1}
I20260927 21:08:43.119825 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.108300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.0.98.1:39611)
0927 21:08:43.109240 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.109244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.110507 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 21:08:43.110595 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.110598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.110616 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.110682 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:08:43.110885 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.112003 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.117611 (+ 5608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.118418 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.118493 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.118548 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.119698 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.119744 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":59,"threads_started":1}
I20260927 21:08:43.120146 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.108007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.98.3:40975)
0927 21:08:43.108960 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.108965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.108976 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:08:43.109015 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.109017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.109033 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.109116 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:08:43.109452 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.113297 (+ 3845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.115706 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.116494 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.116567 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.116640 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.120028 (+ 3388us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.120070 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":99,"threads_started":1}
I20260927 21:08:43.120961 11725 tablet_service.cc:1467] Processing CreateTablet for tablet 1985ebb4493c497c96303f8ed8f750a4 (DEFAULT_TABLE table=test-table [id=0345f263b5ba41d0aa3ae46b4eec16d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:08:43.121336 11725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1985ebb4493c497c96303f8ed8f750a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:08:43.125799 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 1985ebb4493c497c96303f8ed8f750a4 (DEFAULT_TABLE table=test-table [id=0345f263b5ba41d0aa3ae46b4eec16d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:08:43.126132 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1985ebb4493c497c96303f8ed8f750a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:08:43.132035 12102 tablet_bootstrap.cc:492] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Bootstrap starting.
I20260927 21:08:43.133104 12102 tablet_bootstrap.cc:654] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Neither blocks nor log segments found. Creating new log.
I20260927 21:08:43.133330 12101 tablet_bootstrap.cc:492] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da: Bootstrap starting.
I20260927 21:08:43.139482 12101 tablet_bootstrap.cc:654] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da: Neither blocks nor log segments found. Creating new log.
I20260927 21:08:43.141112 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.107887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.0.1:57846)
0927 21:08:43.115572 (+ 7685us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.115581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:43.115624 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:43.130496 (+ 14872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:43.130497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:43.130500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:08:43.130549 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.130551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.133553 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.133554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.134775 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.134778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.140572 (+ 5794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.140574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.140689 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.140697 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:08:43.140749 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:08:43.141014 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7669,"thread_start_us":57,"threads_started":1}
I20260927 21:08:43.141680 12101 log.cc:824] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da: Log is configured to *not* fsync() on all Append() calls
I20260927 21:08:43.142083 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.108018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.0.98.2:36991)
0927 21:08:43.129825 (+ 21807us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.129830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.129843 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:08:43.129900 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.129902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.129919 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.130391 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:08:43.130637 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.131762 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.135151 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.135964 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.141830 (+ 5866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.141921 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.141964 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.142004 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21748,"thread_start_us":95,"threads_started":1}
I20260927 21:08:43.142434 12102 log.cc:824] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:08:43.147174 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 1985ebb4493c497c96303f8ed8f750a4 (DEFAULT_TABLE table=test-table [id=0345f263b5ba41d0aa3ae46b4eec16d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:08:43.147547 11882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1985ebb4493c497c96303f8ed8f750a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:08:43.153959 12105 tablet_bootstrap.cc:492] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Bootstrap starting.
I20260927 21:08:43.154122 12102 tablet_bootstrap.cc:492] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: No bootstrap required, opened a new log
I20260927 21:08:43.154197 12102 ts_tablet_manager.cc:1397] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20260927 21:08:43.155102 12105 tablet_bootstrap.cc:654] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Neither blocks nor log segments found. Creating new log.
I20260927 21:08:43.156360 12102 raft_consensus.cc:348] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.156550 12102 raft_consensus.cc:374] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:08:43.156592 12102 raft_consensus.cc:729] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908fe9a180b0441398c4d908911f3cf3, State: Initialized, Role: FOLLOWER
I20260927 21:08:43.156725 12102 consensus_queue.cc:260] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.156785 12105 log.cc:824] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Log is configured to *not* fsync() on all Append() calls
I20260927 21:08:43.157805 12102 ts_tablet_manager.cc:1428] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:08:43.157929 12082 heartbeater.cc:503] Master 127.0.98.62:38011 was elected leader, sending a full tablet report...
I20260927 21:08:43.159245 12101 tablet_bootstrap.cc:492] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da: No bootstrap required, opened a new log
I20260927 21:08:43.159333 12101 ts_tablet_manager.cc:1397] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20260927 21:08:43.162479 12101 raft_consensus.cc:348] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.163077 12101 raft_consensus.cc:374] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:08:43.163152 12101 raft_consensus.cc:729] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0dba6df1a64ffbad0b351339c8b6da, State: Initialized, Role: FOLLOWER
I20260927 21:08:43.163290 12101 consensus_queue.cc:260] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.163569 12101 ts_tablet_manager.cc:1428] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:08:43.163687 11810 heartbeater.cc:503] Master 127.0.98.62:38011 was elected leader, sending a full tablet report...
I20260927 21:08:43.165418 12105 tablet_bootstrap.cc:492] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: No bootstrap required, opened a new log
I20260927 21:08:43.165557 12105 ts_tablet_manager.cc:1397] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260927 21:08:43.168527 12105 raft_consensus.cc:348] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.168747 12105 raft_consensus.cc:374] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:08:43.168793 12105 raft_consensus.cc:729] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5b20c21f78449a83865f257eddb639, State: Initialized, Role: FOLLOWER
I20260927 21:08:43.168931 12105 consensus_queue.cc:260] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.169224 12105 ts_tablet_manager.cc:1428] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:08:43.169346 11948 heartbeater.cc:503] Master 127.0.98.62:38011 was elected leader, sending a full tablet report...
I20260927 21:08:43.199020 12106 raft_consensus.cc:482] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:08:43.199198 12106 raft_consensus.cc:504] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.199623 12106 leader_election.cc:288] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0dba6df1a64ffbad0b351339c8b6da (127.0.98.1:39611), 5d5b20c21f78449a83865f257eddb639 (127.0.98.2:36991)
I20260927 21:08:43.203962 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.199869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39611 (local address 127.0.98.3:59977)
0927 21:08:43.200168 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.200180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:43.200232 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:43.200647 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:43.200649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:43.200650 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:08:43.200716 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.200719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.201632 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.201633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.202855 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.202858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.203714 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.203715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.203793 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.203797 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:08:43.203848 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:08:43.203890 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260927 21:08:43.204487 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.200362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59977 (local address 127.0.98.1:39611)
0927 21:08:43.200464 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.200467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.200475 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:08:43.200502 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.200503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.200513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.200562 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:08:43.200788 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.201529 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.202977 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.203597 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.203670 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.203681 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.204347 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.204390 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:08:43.204748 11764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1985ebb4493c497c96303f8ed8f750a4" candidate_uuid: "908fe9a180b0441398c4d908911f3cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0dba6df1a64ffbad0b351339c8b6da" is_pre_election: true
I20260927 21:08:43.204906 11764 raft_consensus.cc:2454] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908fe9a180b0441398c4d908911f3cf3 in term 0.
I20260927 21:08:43.205137 11969 leader_election.cc:302] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908fe9a180b0441398c4d908911f3cf3, af0dba6df1a64ffbad0b351339c8b6da; no voters:
I20260927 21:08:43.205272 12106 raft_consensus.cc:2781] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:08:43.205324 12106 raft_consensus.cc:482] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:08:43.205355 12106 raft_consensus.cc:3037] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:08:43.206300 12106 raft_consensus.cc:504] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.206454 12106 leader_election.cc:288] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [CANDIDATE]: Term 1 election: Requested vote from peers af0dba6df1a64ffbad0b351339c8b6da (127.0.98.1:39611), 5d5b20c21f78449a83865f257eddb639 (127.0.98.2:36991)
I20260927 21:08:43.206634 11764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1985ebb4493c497c96303f8ed8f750a4" candidate_uuid: "908fe9a180b0441398c4d908911f3cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0dba6df1a64ffbad0b351339c8b6da"
I20260927 21:08:43.206713 11764 raft_consensus.cc:3037] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:08:43.207640 11764 raft_consensus.cc:2454] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908fe9a180b0441398c4d908911f3cf3 in term 1.
I20260927 21:08:43.207808 11969 leader_election.cc:302] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908fe9a180b0441398c4d908911f3cf3, af0dba6df1a64ffbad0b351339c8b6da; no voters:
I20260927 21:08:43.207878 12106 raft_consensus.cc:2781] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:08:43.208010 12106 raft_consensus.cc:686] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 908fe9a180b0441398c4d908911f3cf3, State: Running, Role: LEADER
I20260927 21:08:43.208099 12106 consensus_queue.cc:237] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } }
I20260927 21:08:43.208920 11598 catalog_manager.cc:5495] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 908fe9a180b0441398c4d908911f3cf3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "908fe9a180b0441398c4d908911f3cf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "908fe9a180b0441398c4d908911f3cf3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } }
I20260927 21:08:43.212893 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.199969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.98.3:42115)
0927 21:08:43.204043 (+ 4074us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.204055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:08:43.204102 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:08:43.209892 (+ 5790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:08:43.209894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:08:43.209896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:08:43.209933 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.209934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.210862 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.210863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.211927 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:08:43.211929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.212687 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.212688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:08:43.212749 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.212753 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:08:43.212789 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 21:08:43.212821 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4062,"thread_start_us":103,"threads_started":1}
I20260927 21:08:43.213152 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.200409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42115 (local address 127.0.98.2:36991)
0927 21:08:43.200615 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.200616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.204153 (+ 3537us) server_negotiation.cc:407] Connection header received
0927 21:08:43.204180 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.204181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.204192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.204233 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:08:43.210033 (+ 5800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.210743 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.212031 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.212574 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.212645 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.212656 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.213046 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.213087 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260927 21:08:43.213459 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1985ebb4493c497c96303f8ed8f750a4" candidate_uuid: "908fe9a180b0441398c4d908911f3cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d5b20c21f78449a83865f257eddb639" is_pre_election: true
I20260927 21:08:43.213593 11899 raft_consensus.cc:2454] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908fe9a180b0441398c4d908911f3cf3 in term 0.
I20260927 21:08:43.213693 11898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1985ebb4493c497c96303f8ed8f750a4" candidate_uuid: "908fe9a180b0441398c4d908911f3cf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d5b20c21f78449a83865f257eddb639"
I20260927 21:08:43.213740 11898 raft_consensus.cc:3037] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:08:43.214581 11898 raft_consensus.cc:2454] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908fe9a180b0441398c4d908911f3cf3 in term 1.
I20260927 21:08:43.238857 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.235616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.98.3:40975)
0927 21:08:43.235657 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:08:43.235659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:08:43.235666 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:08:43.235704 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:08:43.235704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:08:43.235724 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:08:43.235787 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:08:43.235989 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.236670 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.237740 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:08:43.238060 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:08:43.238122 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:08:43.238320 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:08:43.238450 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:08:43.238524 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:08:43.238742 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:08:43.238762 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:08:43.240870 11898 raft_consensus.cc:1261] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 1 FOLLOWER]: Refusing update from remote peer 908fe9a180b0441398c4d908911f3cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:08:43.241297 12111 consensus_queue.cc:1035] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d5b20c21f78449a83865f257eddb639" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:08:43.241499 11764 raft_consensus.cc:1261] T 1985ebb4493c497c96303f8ed8f750a4 P af0dba6df1a64ffbad0b351339c8b6da [term 1 FOLLOWER]: Refusing update from remote peer 908fe9a180b0441398c4d908911f3cf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:08:43.242822 12111 consensus_queue.cc:1035] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0dba6df1a64ffbad0b351339c8b6da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:08:43.248728 11598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:08:43.248819 11598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:08:43.249471 11598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc31f098649f43189e5000cb643638c9: Sending DeleteTablet for 3 replicas of tablet 1985ebb4493c497c96303f8ed8f750a4
I20260927 21:08:43.250092 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 1985ebb4493c497c96303f8ed8f750a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:08:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37660
I20260927 21:08:43.250269 11725 tablet_service.cc:1514] Processing DeleteTablet for tablet 1985ebb4493c497c96303f8ed8f750a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:08:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40452
I20260927 21:08:43.250516 11882 tablet_service.cc:1514] Processing DeleteTablet for tablet 1985ebb4493c497c96303f8ed8f750a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:08:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57846
I20260927 21:08:43.250898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 11679
I20260927 21:08:43.252118 12122 tablet_replica.cc:330] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: stopping tablet replica
I20260927 21:08:43.252218 12122 raft_consensus.cc:2229] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:08:43.252310 12122 raft_consensus.cc:2258] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:08:43.252523 12115 mvcc.cc:205] Tried to move back new op lower bound from 7334065451991707648 to 7334065451860946944. Current Snapshot: MvccSnapshot[applied={T|T < 7334065451991707648}]
Sep 27 21:08:43 dist-test-slave-8bgs krb5kdc[11122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:08:43.259599 12122 ts_tablet_manager.cc:1905] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:08:43.260071 12124 tablet_replica.cc:330] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: stopping tablet replica
I20260927 21:08:43.260176 12124 raft_consensus.cc:2229] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:08:43.260274 12124 raft_consensus.cc:2258] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:08:43.262557 12124 ts_tablet_manager.cc:1905] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:08:43.264509 12124 ts_tablet_manager.cc:1918] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:08:43.264600 12124 log.cc:1196] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/1985ebb4493c497c96303f8ed8f750a4
I20260927 21:08:43.265167 12124 ts_tablet_manager.cc:1939] T 1985ebb4493c497c96303f8ed8f750a4 P 5d5b20c21f78449a83865f257eddb639: Deleting consensus metadata
I20260927 21:08:43.265561 11582 catalog_manager.cc:4841] TS 5d5b20c21f78449a83865f257eddb639 (127.0.98.2:36991): tablet 1985ebb4493c497c96303f8ed8f750a4 (table test-table [id=0345f263b5ba41d0aa3ae46b4eec16d5]) successfully deleted
I20260927 21:08:43.265707 12122 ts_tablet_manager.cc:1918] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:08:43.265766 12122 log.cc:1196] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/1985ebb4493c497c96303f8ed8f750a4
I20260927 21:08:43.266013 12122 ts_tablet_manager.cc:1939] T 1985ebb4493c497c96303f8ed8f750a4 P 908fe9a180b0441398c4d908911f3cf3: Deleting consensus metadata
I20260927 21:08:43.266350 11582 catalog_manager.cc:4841] TS 908fe9a180b0441398c4d908911f3cf3 (127.0.98.3:40975): tablet 1985ebb4493c497c96303f8ed8f750a4 (table test-table [id=0345f263b5ba41d0aa3ae46b4eec16d5]) successfully deleted
W20260927 21:08:43.266469 11582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39611 (error 108)
I20260927 21:08:43.268626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 11816
I20260927 21:08:43.271986 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:08:43.271826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39611 (local address 127.0.0.1:40454)
0927 21:08:43.271879 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:08:43.271930 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 21:08:43.272101 11582 catalog_manager.cc:4584] TS af0dba6df1a64ffbad0b351339c8b6da (127.0.98.1:39611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1985ebb4493c497c96303f8ed8f750a4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39611: connect: Connection refused (error 111)
I20260927 21:08:43.275725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 11953
I20260927 21:08:43.281219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 11566
I20260927 21:08:43.288275 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:08:43.632470 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:08:43.632606 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:08:43.976652 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:08:43.976 UTC [11152] LOG: received smart shutdown request
2026-09-27 21:08:43.980 UTC [11152] LOG: background worker "logical replication launcher" (PID 11161) exited with exit code 1
2026-09-27 21:08:43.980 UTC [11156] LOG: shutting down
2026-09-27 21:08:44.141 UTC [11152] LOG: database system is shut down
2026-09-27T21:08:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45602 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:08:44 dist-test-slave-8bgs krb5kdc[12135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:08:44 dist-test-slave-8bgs krb5kdc[12135](info): set up 2 sockets
Sep 27 21:08:44 dist-test-slave-8bgs krb5kdc[12135](info): commencing operation
krb5kdc: starting...
W20260927 21:08:45.299160 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:08:45 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:08:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 21:08:45.458169 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6L0Aex/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/postgres -l logfile start
2026-09-27 21:08:47.831 UTC [12165] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:08:47.831 UTC [12165] LOG: listening on IPv4 address "127.0.98.20", port 48423
2026-09-27 21:08:47.834 UTC [12165] LOG: listening on Unix socket "/tmp/.s.PGSQL.48423"
2026-09-27 21:08:47.868 UTC [12168] LOG: database system was shut down at 2026-09-27 21:08:46 UTC
2026-09-27 21:08:47.872 UTC [12165] LOG: database system is ready to accept connections
I20260927 21:08:48.814899 392 mini_postgres.cc:96] Postgres bound to 48423
2026-09-27 21:08:48.821 UTC [12176] FATAL: database "slave" does not exist
127.0.98.20:48423 - accepting connections
I20260927 21:08:48.821815 392 mini_ranger.cc:161] Starting Ranger...
I20260927 21:08:48.836114 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:08:49.287837 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:08:49.287914 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-admin
2026-09-27 21:08:49,553 [I] DB FLAVOR :POSTGRES
2026-09-27 21:08:49,553 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:08:49,553 [I] Checking connection..
2026-09-27 21:08:49,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:08:49,789 [I] Checking connection passed.
2026-09-27 21:08:49,789 [I] --------- Verifying version history table ---------
2026-09-27 21:08:49,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:08:50,018 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:08:50,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:08:50,263 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:08:50,263 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:08:50,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:08:50.503 UTC [12240] WARNING: there is no transaction in progress
2026-09-27 21:08:50,509 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:08:50,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:08:50,758 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:08:50,758 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:08:50,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:08:51,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:08:51,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8bgs.c.gcp-upstream.internal','N') ;"
2026-09-27 21:08:51,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:08:51,748 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:08:51,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:08:51,974 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:08:51,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:08:52,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:08:52,477 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:08:52,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:08:53.397 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.413 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.428 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.442 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.460 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.480 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.536 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.543 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.556 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.565 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.575 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.587 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.596 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.603 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.609 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:53.614 UTC [12348] WARNING: there is no transaction in progress
2026-09-27 21:08:54,188 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:08:54,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:08:54,432 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:08:54,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:08:54,667 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:08:54,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:08:54,911 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:08:54,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:08:55,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:08:55,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8bgs.c.gcp-upstream.internal';"
2026-09-27 21:08:55,599 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:08:55,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:08:55,830 [I] DB_PATCHES have already been applied
I20260927 21:08:55.836037 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6L0Aex/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:08:55.836105 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 21:08:55.839215 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54991
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:08:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:08:56.953285 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.131s user 0.000s sys 0.008s
Sep 27, 2026 9:09:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:09:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 9:09:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:09:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:09:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:09:18.932897 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:09:19 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543359, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:09:19.038097 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:09:19.050098 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:09:19.875455 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:09:19.875542 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms
I20260927 21:09:19.875564 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:48423
2026-09-27 21:09:20,037 [I] DB FLAVOR :POSTGRES
2026-09-27 21:09:20,037 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:09:20,038 [I] Checking connection
2026-09-27 21:09:20,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:09:20,262 [I] connection success
2026-09-27 21:09:20,263 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:09:20,263 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:09:20,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:09:20,507 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:09:20,508 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:09:20,508 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:09:20,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:09:20,753 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:09:21.239815 392 mini_ranger_kms.cc:310] Created kms service
I20260927 21:09:21.429962 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:09:21.475627 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:09:21 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543359, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:21.869323 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:09:21.869437 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task6L0Aex/build/release/bin/postgresql.jar:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6L0Aex/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:09:21.869457 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 21:09:21.873670 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12528
I20260927 21:09:21.873767 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54411
Sep 27, 2026 9:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:09:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:09:24.158823 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.121s user 0.000s sys 0.145s
I20260927 21:09:24.158926 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:09:24.158982 392 mini_ranger_kms.cc:401] 127.0.98.20:54411/kms/v1/keys
Sep 27 21:09:24 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543364, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:09:24 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543364, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:09:25 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543364, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 21:09:27 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:09:27.321743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40119
--ranger_config_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:27.406455 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:27.406698 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:27.406735 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:27.407980 12579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:27.408051 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:27.408073 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:27.408104 12579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:27.408152 12579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:27.410228 12579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40119
--ranger_config_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:27.410514 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:27.410825 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:27.416129 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:27.416136 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:27.416354 12579 server_base.cc:1034] running on GCE node
W20260927 21:09:27.416145 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:27.416754 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:27.417109 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:27.418298 12579 hybrid_clock.cc:648] HybridClock initialized: now 1790543367418270 us; error 41 us; skew 500 ppm
Sep 27 21:09:27 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:27.424679 12579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:27.425226 12579 webserver.cc:460] Webserver started at http://127.0.98.62:44317/ using document root <none> and password file <none>
I20260927 21:09:27.425485 12579 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:27.425544 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:27.425688 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:09:27 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:27.477352 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "07baf5ec6c334e289390d45a47e3b5c6"
format_stamp: "Formatted at 2026-09-27 21:09:27 on dist-test-slave-8bgs"
server_key: "a113bd1c72a0a26dcb2231f5225d13db"
server_key_iv: "6ad1583a9ad9544f7b988182e5cf31e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:27.477788 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "07baf5ec6c334e289390d45a47e3b5c6"
format_stamp: "Formatted at 2026-09-27 21:09:27 on dist-test-slave-8bgs"
server_key: "a113bd1c72a0a26dcb2231f5225d13db"
server_key_iv: "6ad1583a9ad9544f7b988182e5cf31e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:27.479357 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:09:27.494645 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:27.494925 12579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:27.495048 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "07baf5ec6c334e289390d45a47e3b5c6"
format_stamp: "Formatted at 2026-09-27 21:09:27 on dist-test-slave-8bgs"
server_key: "a113bd1c72a0a26dcb2231f5225d13db"
server_key_iv: "6ad1583a9ad9544f7b988182e5cf31e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:27.495182 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:27.509279 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:27.511404 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:27.511561 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:27.515938 12579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40119
I20260927 21:09:27.515978 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40119 every 8 connection(s)
I20260927 21:09:27.516367 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:27.517036 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:27.517799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 12579
I20260927 21:09:27.517922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:09:27.518112 392 external_mini_cluster.cc:1428] Setting key 8b399736588a8847e1081bdf087739f1
I20260927 21:09:27.519833 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Bootstrap starting.
I20260927 21:09:27.520594 12647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:27.520931 12647 log.cc:824] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:27.521610 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: No bootstrap required, opened a new log
I20260927 21:09:27.523912 12647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:27.524111 12647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:27.524178 12647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07baf5ec6c334e289390d45a47e3b5c6, State: Initialized, Role: FOLLOWER
I20260927 21:09:27.524333 12647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:27.524420 12647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:27.524454 12647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:27.524538 12647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:27.525303 12647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:27.525422 12647 leader_election.cc:302] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07baf5ec6c334e289390d45a47e3b5c6; no voters:
I20260927 21:09:27.525583 12647 leader_election.cc:288] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:09:27.525635 12652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:27.525876 12652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 LEADER]: Becoming Leader. State: Replica: 07baf5ec6c334e289390d45a47e3b5c6, State: Running, Role: LEADER
I20260927 21:09:27.525976 12647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:27.526014 12652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
Sep 27 21:09:27 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:27.526363 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07baf5ec6c334e289390d45a47e3b5c6. Latest consensus state: current_term: 1 leader_uuid: "07baf5ec6c334e289390d45a47e3b5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } } }
I20260927 21:09:27.526458 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:27.526365 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07baf5ec6c334e289390d45a47e3b5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } } }
I20260927 21:09:27.526791 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:27.526928 12656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:27.527460 12656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:27.527954 12647 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:09:27.533155 12656 catalog_manager.cc:1347] Generated new cluster ID: 78c879530faf40e4a36c3080bc63a0dc
I20260927 21:09:27.533339 12656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:09:27.534576 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:27.518655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.98.62:40119)
0927 21:09:27.519207 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:27.519217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:27.519251 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:09:27.519792 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:27.519802 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:27.519866 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:27.520021 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:27.520794 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:27.522173 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:27.523354 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:27.523956 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:27.524070 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:27.526919 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:27.526926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:27.526927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:27.526939 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:27.531475 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:27.532065 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:27.532069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:27.532070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:27.532289 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:27.532560 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:27.532561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:27.532562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:27.533132 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:27.533382 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:27.533879 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 21:09:27.534275 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":77,"threads_started":1}
I20260927 21:09:27.553082 12656 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:09:27.554015 12656 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:27.566956 12656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Generated new TSK 0
I20260927 21:09:27.567297 12656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:09:28.258612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:28.384624 12692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:28.384914 12692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:28.384959 12692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:28.386700 12692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:28.386759 12692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:28.386803 12692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:09:28.389063 12692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:28.389304 12692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:28.389585 12692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:28.395112 12700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:28.395159 12697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:28.395160 12698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:28.395468 12692 server_base.cc:1034] running on GCE node
I20260927 21:09:28.395699 12692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:28.395955 12692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:28.397122 12692 hybrid_clock.cc:648] HybridClock initialized: now 1790543368397109 us; error 32 us; skew 500 ppm
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:28.399753 12692 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:09:28.400336 12692 webserver.cc:460] Webserver started at http://127.0.98.1:33071/ using document root <none> and password file <none>
I20260927 21:09:28.400591 12692 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:28.400660 12692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:28.400794 12692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:28.414680 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "6cdfa48e94824444900b2efb1fb890ff"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "7eacb365df34b565d8f7cef5c78ab835"
server_key_iv: "37734894d4faf5b77e94daab8bb626e0"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.415607 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6cdfa48e94824444900b2efb1fb890ff"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "7eacb365df34b565d8f7cef5c78ab835"
server_key_iv: "37734894d4faf5b77e94daab8bb626e0"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.417621 12692 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:09:28.439291 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.439568 12692 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 21:09:28.439703 12692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "6cdfa48e94824444900b2efb1fb890ff"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "7eacb365df34b565d8f7cef5c78ab835"
server_key_iv: "37734894d4faf5b77e94daab8bb626e0"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.439893 12692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:28.466238 12692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:28.468431 12692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:28.468608 12692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:28.468873 12692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:28.469254 12692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:28.469304 12692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.469372 12692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:28.469416 12692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.476733 12692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35871
I20260927 21:09:28.477109 12692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:09:28.477705 12822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35871 every 8 connection(s)
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:28.485898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 12692
I20260927 21:09:28.486032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:09:28.486228 392 external_mini_cluster.cc:1428] Setting key 5486994ff51e9f4ff2dde4dfeda0921f
I20260927 21:09:28.486753 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:28.478216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41455 (local address 127.0.98.62:40119)
0927 21:09:28.478350 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:28.478352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:28.478363 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:09:28.478775 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:28.478776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:28.478791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:28.478866 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:28.479496 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.480022 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.481054 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.481289 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.481376 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:28.483454 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:28.483455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:28.483456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:28.483464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:28.485153 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:28.485561 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:28.485562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:28.485563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:28.485633 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:28.485976 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:28.485977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:28.485977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:28.486125 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:28.486204 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:28.486561 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 21:09:28.486648 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260927 21:09:28.486945 12825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:28.477872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.1:41455)
0927 21:09:28.478043 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:28.478087 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:28.478666 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:28.478944 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:28.478947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:28.479203 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:28.479405 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:28.479409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.480119 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.480121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:28.480972 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:28.480975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.481397 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.481398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:28.481518 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:28.482056 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:28.482065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:28.483357 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:28.485254 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:28.485260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:28.485261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:28.485480 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:28.485740 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:28.485741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:28.485741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:28.485894 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:28.486208 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:28.486214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:28.486398 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 21:09:28.486534 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:09:28.486706 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":103,"threads_started":1}
I20260927 21:09:28.489059 12823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260927 21:09:28.489176 12823 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:28.489372 12823 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:28.489944 12611 ts_manager.cc:194] Registered new tserver with Master: 6cdfa48e94824444900b2efb1fb890ff (127.0.98.1:35871)
I20260927 21:09:28.490823 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41455
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:09:28.522516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:09:28.643177 12829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:28.643471 12829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:28.643508 12829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:28.645288 12829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:28.645351 12829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:28.645390 12829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:09:28.647574 12829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:28.647810 12829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:28.648144 12829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:28.654188 12839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:28.654254 12836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:28.654531 12829 server_base.cc:1034] running on GCE node
W20260927 21:09:28.654733 12837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:28.654987 12829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:28.655289 12829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:28.656469 12829 hybrid_clock.cc:648] HybridClock initialized: now 1790543368656439 us; error 42 us; skew 500 ppm
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:28.659659 12829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:09:28.660166 12829 webserver.cc:460] Webserver started at http://127.0.98.2:39163/ using document root <none> and password file <none>
I20260927 21:09:28.660408 12829 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:28.660466 12829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:28.660590 12829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:28.674036 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "9399fe230b594fa6860f55024b2bb12e"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "dc189e961b70a052ca3abcbff3b0b386"
server_key_iv: "38ea090830c1057486f498fd3217a247"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.674499 12829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9399fe230b594fa6860f55024b2bb12e"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "dc189e961b70a052ca3abcbff3b0b386"
server_key_iv: "38ea090830c1057486f498fd3217a247"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.679955 12829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:09:28.700886 12847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.701844 12829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260927 21:09:28.701972 12829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "9399fe230b594fa6860f55024b2bb12e"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "dc189e961b70a052ca3abcbff3b0b386"
server_key_iv: "38ea090830c1057486f498fd3217a247"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.702082 12829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:28.723711 12829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:28.727059 12829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:28.727252 12829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:28.727499 12829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:28.727871 12829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:28.727922 12829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.727962 12829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:28.727984 12829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.734545 12829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44107
I20260927 21:09:28.734994 12829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:09:28.735248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 12829
I20260927 21:09:28.735369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:09:28.735581 392 external_mini_cluster.cc:1428] Setting key f632b4bc315a8a78e0109695d99a99ac
I20260927 21:09:28.740411 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44107 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 21:09:28.765259 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:28.751529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48291 (local address 127.0.98.62:40119)
0927 21:09:28.751586 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:28.751588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:28.751768 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:09:28.752498 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:28.752500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:28.752512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:28.752545 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:28.753420 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.754176 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.755770 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.756063 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.756111 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:28.759375 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:28.759376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:28.759377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:28.759384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:28.761843 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:28.762401 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:28.762402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:28.762403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:28.762468 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:28.762799 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:28.762800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:28.762801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:28.762956 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:28.763021 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:28.765042 (+ 2021us) server_negotiation.cc:299] Negotiation successful
0927 21:09:28.765145 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:09:28.765766 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:28.751187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.2:48291)
0927 21:09:28.751623 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:28.751673 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:28.752364 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:28.752749 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:28.752753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:28.753003 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:28.753300 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:28.753308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.754309 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.754310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:28.755671 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:28.755674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:28.756230 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:28.756231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:28.756374 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:28.757194 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:28.757205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:28.759258 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:28.761947 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:28.761955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:28.761957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:28.762281 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:28.762543 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:28.762544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:28.762544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:28.762711 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:28.763025 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:28.763031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:28.763250 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 21:09:28.765360 (+ 2110us) client_negotiation.cc:240] Negotiation successful
0927 21:09:28.765534 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":114,"threads_started":1}
I20260927 21:09:28.766906 12962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260927 21:09:28.767027 12962 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:28.767290 12962 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:28.767678 12611 ts_manager.cc:194] Registered new tserver with Master: 9399fe230b594fa6860f55024b2bb12e (127.0.98.2:44107)
I20260927 21:09:28.768138 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:09:28.794884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:28.908766 12966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:28.909054 12966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:28.909090 12966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:28.910765 12966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:28.910825 12966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:28.910864 12966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:09:28.913000 12966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:28.913264 12966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:28.913523 12966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:28.919652 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:28.920006 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:28.920233 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:28.921192 12966 server_base.cc:1034] running on GCE node
I20260927 21:09:28.921444 12966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:28.921772 12966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:28.923007 12966 hybrid_clock.cc:648] HybridClock initialized: now 1790543368922986 us; error 36 us; skew 500 ppm
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:28.931900 12966 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:09:28.932493 12966 webserver.cc:460] Webserver started at http://127.0.98.3:43317/ using document root <none> and password file <none>
I20260927 21:09:28.932785 12966 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:28.932852 12966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:28.932992 12966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:09:28 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:28.946938 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "f22ba94e179b42d08bbcf8304c6e048f"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "bfdda87eb467ca4315e9b738b75c8f3d"
server_key_iv: "fa14cdcb1ba5bc3235db675d4a39f9b1"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.947544 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f22ba94e179b42d08bbcf8304c6e048f"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "bfdda87eb467ca4315e9b738b75c8f3d"
server_key_iv: "fa14cdcb1ba5bc3235db675d4a39f9b1"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.955476 12966 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260927 21:09:28.981153 12981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:28.981367 12966 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260927 21:09:28.981465 12966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "f22ba94e179b42d08bbcf8304c6e048f"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "bfdda87eb467ca4315e9b738b75c8f3d"
server_key_iv: "fa14cdcb1ba5bc3235db675d4a39f9b1"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:28.981580 12966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:29.001878 12966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:29.004243 12966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:29.004439 12966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:29.004696 12966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:29.005056 12966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:29.005108 12966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.005147 12966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:29.005185 12966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.012590 12966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46337
I20260927 21:09:29.013031 12966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:09:29.015618 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46337 every 8 connection(s)
I20260927 21:09:29.022805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 12966
I20260927 21:09:29.022935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:09:29.023157 392 external_mini_cluster.cc:1428] Setting key 95f782549e4de0693fc39d129d76a517
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:29.029690 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.016242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37975 (local address 127.0.98.62:40119)
0927 21:09:29.016288 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:29.016291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:29.019428 (+ 3137us) server_negotiation.cc:407] Connection header received
0927 21:09:29.020032 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:29.020034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:29.020047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:29.020079 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:29.021336 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.021846 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.023069 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.023420 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.023485 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.026084 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:29.026085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:29.026086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:29.026093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:29.027927 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.028417 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.028418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.028418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.028476 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.028789 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.028791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.028791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.028932 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:29.028975 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:29.029489 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 21:09:29.029582 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:09:29.030210 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.016250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.3:37975)
0927 21:09:29.019272 (+ 3022us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:29.019326 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:29.019905 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:29.020273 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:29.020277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:29.020987 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:29.021235 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.021240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.021965 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.021966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.022958 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.022961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.023600 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.023601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.023746 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.024341 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:29.024360 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:29.025969 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:29.028018 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.028025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.028027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.028325 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.028548 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.028549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.028549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.028710 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.029021 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:29.029027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:29.029201 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 21:09:29.029802 (+ 601us) client_negotiation.cc:240] Negotiation successful
0927 21:09:29.029977 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2958,"thread_start_us":84,"threads_started":1}
I20260927 21:09:29.030742 13095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260927 21:09:29.030880 13095 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.031067 13095 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:29.031498 12611 ts_manager.cc:194] Registered new tserver with Master: f22ba94e179b42d08bbcf8304c6e048f (127.0.98.3:46337)
I20260927 21:09:29.032011 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37975
I20260927 21:09:29.035051 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:09:29.035621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 12692
I20260927 21:09:29.047341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 12829
I20260927 21:09:29.053987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 12966
I20260927 21:09:29.060869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 12579
I20260927 21:09:29.067471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40119
--webserver_interface=127.0.98.62
--webserver_port=44317
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40119
--ranger_config_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:29.164177 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:29.164382 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:29.164404 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:29.165467 13098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:29.165500 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:29.165514 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:29.165526 13098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:29.165539 13098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:29.166987 13098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40119
--ranger_config_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:29.167208 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:29.167387 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:29.172029 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:29.172058 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:29.172027 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:29.172268 13098 server_base.cc:1034] running on GCE node
I20260927 21:09:29.172534 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:29.172827 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:29.173985 13098 hybrid_clock.cc:648] HybridClock initialized: now 1790543369173969 us; error 35 us; skew 500 ppm
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:29.176374 13098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:29.176846 13098 webserver.cc:460] Webserver started at http://127.0.98.62:44317/ using document root <none> and password file <none>
I20260927 21:09:29.177054 13098 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:29.177126 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:29.247861 13098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:09:29.248770 13114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.248975 13098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:09:29.249224 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "07baf5ec6c334e289390d45a47e3b5c6"
format_stamp: "Formatted at 2026-09-27 21:09:27 on dist-test-slave-8bgs"
server_key: "a113bd1c72a0a26dcb2231f5225d13db"
server_key_iv: "6ad1583a9ad9544f7b988182e5cf31e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:29.249686 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:29.269424 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:29.271492 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:29.271605 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:29.275656 13098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40119
I20260927 21:09:29.275696 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40119 every 8 connection(s)
I20260927 21:09:29.276049 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:29.277163 13167 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:09:29.277808 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Bootstrap starting.
I20260927 21:09:29.283651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13098
I20260927 21:09:29.284029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40119
--webserver_port=33071
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:29.286281 13167 log.cc:824] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:29.288183 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:29.288477 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Bootstrap complete.
I20260927 21:09:29.290848 13167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:29.291069 13167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07baf5ec6c334e289390d45a47e3b5c6, State: Initialized, Role: FOLLOWER
I20260927 21:09:29.291317 13167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:29.291383 13167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:29.291440 13167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:29.291487 13167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:09:29.292498 13167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:29.292588 13167 leader_election.cc:302] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07baf5ec6c334e289390d45a47e3b5c6; no voters:
I20260927 21:09:29.292788 13167 leader_election.cc:288] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:09:29.292829 13171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:09:29.292941 13171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [term 2 LEADER]: Becoming Leader. State: Replica: 07baf5ec6c334e289390d45a47e3b5c6, State: Running, Role: LEADER
I20260927 21:09:29.293025 13171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } }
I20260927 21:09:29.293087 13167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:29.293267 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07baf5ec6c334e289390d45a47e3b5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } } }
I20260927 21:09:29.293339 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:29.293445 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07baf5ec6c334e289390d45a47e3b5c6. Latest consensus state: current_term: 2 leader_uuid: "07baf5ec6c334e289390d45a47e3b5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07baf5ec6c334e289390d45a47e3b5c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40119 } } }
I20260927 21:09:29.293491 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:29.293607 13175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:29.294116 13175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:29.294265 13175 catalog_manager.cc:1259] Loaded cluster ID: 78c879530faf40e4a36c3080bc63a0dc
I20260927 21:09:29.294305 13175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:09:29.294570 13167 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:09:29.297993 13175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:29.298463 13175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Loaded TSK: 0
I20260927 21:09:29.298789 13175 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:09:29.397802 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:29.398070 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:29.398110 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:29.399828 13169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:29.399890 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:29.399930 13169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:09:29.402011 13169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33071
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:29.402257 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:29.402513 13169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:29.409252 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:29.409354 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:29.409484 13169 server_base.cc:1034] running on GCE node
W20260927 21:09:29.409265 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:29.409778 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:29.410056 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:29.411229 13169 hybrid_clock.cc:648] HybridClock initialized: now 1790543369411200 us; error 44 us; skew 500 ppm
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:29.414306 13169 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:09:29.414892 13169 webserver.cc:460] Webserver started at http://127.0.98.1:33071/ using document root <none> and password file <none>
I20260927 21:09:29.415244 13169 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:29.415383 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:29.438082 13169 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260927 21:09:29.438958 13215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.439131 13169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:09:29.439224 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "6cdfa48e94824444900b2efb1fb890ff"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "7eacb365df34b565d8f7cef5c78ab835"
server_key_iv: "37734894d4faf5b77e94daab8bb626e0"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:29.439739 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:29.480568 13169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:29.488785 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:29.489014 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:29.489317 13169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:29.489779 13169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:29.489874 13169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.489981 13169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:29.490056 13169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.496563 13169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35871
I20260927 21:09:29.497071 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:09:29.499135 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35871 every 8 connection(s)
I20260927 21:09:29.506865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13169
I20260927 21:09:29.507354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44107
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40119
--webserver_port=39163
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:29.514488 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.499710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.1:36151)
0927 21:09:29.499991 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:29.500036 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:29.500735 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:29.502233 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:29.502237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:29.502491 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:29.503200 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.503207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.505071 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.505072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.506372 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.506375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.507276 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.507277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.507416 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.508052 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:29.508063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:29.509665 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:29.512367 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.512375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.512377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.512729 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.513046 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.513047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.513048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.513187 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.513719 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:29.513726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:29.513910 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 21:09:29.514013 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:09:29.514190 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20260927 21:09:29.515663 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.501359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36151 (local address 127.0.98.62:40119)
0927 21:09:29.501658 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:29.501668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:29.501712 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:09:29.501934 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:29.501942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:29.502012 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:29.503343 (+ 1331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:29.503439 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.504937 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.506492 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.507153 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.507690 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.509945 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:29.509950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:29.509951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:29.509964 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:29.512260 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.512838 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.512841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.512842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.512958 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.513273 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.513274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.513274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.513595 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:29.515179 (+ 1584us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:29.515263 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:09:29.515400 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260927 21:09:29.516635 13329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260927 21:09:29.516734 13329 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.516947 13329 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
W20260927 21:09:29.517900 13329 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40119 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:09:29.517972 13329 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.518054 13329 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:29.518465 13329 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.518543 13329 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
W20260927 21:09:29.518940 13329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:09:29.613965 13333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:29.614239 13333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:29.614279 13333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:29.615583 13333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:29.615657 13333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:29.615705 13333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:09:29.617357 13333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39163
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:29.617578 13333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:29.617852 13333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:29.623020 13341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:29.623121 13339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:29.623211 13338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:29.623658 13333 server_base.cc:1034] running on GCE node
I20260927 21:09:29.623884 13333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:29.624166 13333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:29.625349 13333 hybrid_clock.cc:648] HybridClock initialized: now 1790543369625336 us; error 27 us; skew 500 ppm
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:29.627928 13333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:09:29.628425 13333 webserver.cc:460] Webserver started at http://127.0.98.2:39163/ using document root <none> and password file <none>
I20260927 21:09:29.628655 13333 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:29.628737 13333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:29.641232 13333 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:09:29.642077 13349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.642234 13333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:29.642320 13333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "9399fe230b594fa6860f55024b2bb12e"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "dc189e961b70a052ca3abcbff3b0b386"
server_key_iv: "38ea090830c1057486f498fd3217a247"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:29.642784 13333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:29.668551 13333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:29.671028 13333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:29.671216 13333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:29.671474 13333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:29.671854 13333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:29.671907 13333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.671939 13333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:29.671963 13333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.678730 13333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44107
I20260927 21:09:29.679028 13462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44107 every 8 connection(s)
I20260927 21:09:29.679888 13333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:09:29.684640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13333
I20260927 21:09:29.684959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46337
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40119
--webserver_port=43317
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:29.691710 13465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.681179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.2:39987)
0927 21:09:29.681321 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:29.681361 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:29.681884 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:29.682142 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:29.682146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:29.682413 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:29.682713 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.682721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.683780 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.683781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.685061 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.685064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.685630 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.685631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.685763 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.686350 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:29.686361 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:29.687992 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:29.689910 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.689918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.689921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.690237 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.690485 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.690486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.690487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.690626 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.690970 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:29.690977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:29.691182 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 21:09:29.691302 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:09:29.691469 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260927 21:09:29.692220 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.681157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39987 (local address 127.0.98.62:40119)
0927 21:09:29.681213 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:29.681216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:29.681460 (+ 244us) server_negotiation.cc:407] Connection header received
0927 21:09:29.682005 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:29.682007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:29.682023 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:29.682073 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:29.682837 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.683660 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.685165 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.685484 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.685529 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.688102 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:29.688103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:29.688104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:29.688111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:29.689800 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.690341 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.690342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.690342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.690409 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.690708 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.690709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.690710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.690870 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:29.690940 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:29.692034 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0927 21:09:29.692128 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:09:29.693038 13463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260927 21:09:29.693136 13463 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.693337 13463 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
W20260927 21:09:29.693956 13463 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40119 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:09:29.694031 13463 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.694098 13463 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:29.694409 13463 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.694480 13463 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
W20260927 21:09:29.694768 13463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:09:29.793932 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:29.794221 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:29.794268 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:29.796022 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:29.796078 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:29.796125 13466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:09:29.797936 13466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43317
--tserver_master_addrs=127.0.98.62:40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:29.798213 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:29.798456 13466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:29.803339 13475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:29.803390 13466 server_base.cc:1034] running on GCE node
W20260927 21:09:29.803378 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:29.803635 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:29.803871 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:29.804080 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:29.805240 13466 hybrid_clock.cc:648] HybridClock initialized: now 1790543369805226 us; error 29 us; skew 500 ppm
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:29.808029 13466 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:09:29.808564 13466 webserver.cc:460] Webserver started at http://127.0.98.3:43317/ using document root <none> and password file <none>
I20260927 21:09:29.808799 13466 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:29.808880 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:29.824591 13466 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:09:29.826086 13483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.826265 13466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:29.826376 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "f22ba94e179b42d08bbcf8304c6e048f"
format_stamp: "Formatted at 2026-09-27 21:09:28 on dist-test-slave-8bgs"
server_key: "bfdda87eb467ca4315e9b738b75c8f3d"
server_key_iv: "fa14cdcb1ba5bc3235db675d4a39f9b1"
server_key_version: "kuduclusterkey@0"
I20260927 21:09:29.826735 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:29.840431 13466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:29.842743 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:29.842907 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:29.843221 13466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:29.843580 13466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:29.843637 13466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.843675 13466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:29.843701 13466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:29.849962 13466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46337
I20260927 21:09:29.850401 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:09:29.850392 13596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46337 every 8 connection(s)
I20260927 21:09:29.855012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13466
Sep 27 21:09:29 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:29.862885 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.851164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40119 (local address 127.0.98.3:33259)
0927 21:09:29.851338 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:29.851379 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:29.851881 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:29.852156 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:29.852160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:29.852348 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:29.852595 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.852601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.853571 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.853572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.854598 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:29.854601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.856131 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.856132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:29.856266 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.856814 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:29.856824 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:29.858491 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:29.861067 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.861075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.861078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.861405 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.861667 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:29.861668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:29.861668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:29.861823 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:29.862103 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:29.862109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:29.862293 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 21:09:29.862426 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 21:09:29.862616 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260927 21:09:29.863399 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.851154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33259 (local address 127.0.98.62:40119)
0927 21:09:29.851205 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:29.851207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:29.851460 (+ 253us) server_negotiation.cc:407] Connection header received
0927 21:09:29.851992 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:29.851994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:29.852018 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:29.852067 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:29.852709 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.853466 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.855684 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.856021 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.856110 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.859193 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:29.859195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:29.859195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:29.859203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:29.860964 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.861513 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.861514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.861514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.861590 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.861907 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.861908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.861908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.862029 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:29.862112 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:29.863248 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 21:09:29.863331 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":21}
I20260927 21:09:29.863761 13597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40119
I20260927 21:09:29.863868 13597 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.864055 13597 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
W20260927 21:09:29.864699 13597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40119 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:09:29.864782 13597 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.864857 13597 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:29.865216 13597 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:29.865296 13597 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
W20260927 21:09:29.865588 13597 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:09:29.867000 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:29.855544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.98.62:40119)
0927 21:09:29.855839 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:29.855841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:29.855855 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:09:29.855873 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:29.855874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:29.855884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:29.855946 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:29.856722 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.859530 (+ 2808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.861881 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:29.862181 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:29.862252 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:29.863860 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:29.863862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:29.863862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:29.863870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:29.865734 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.866140 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.866141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.866141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.866205 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:29.866438 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:29.866439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:29.866439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:29.866574 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:29.866754 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:29.866826 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:09:29.866910 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":80,"threads_started":1}
Sep 27 21:09:30 dist-test-slave-8bgs krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:30.519124 13329 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:30.519282 13329 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:30.519646 13131 ts_manager.cc:194] Registered new tserver with Master: 6cdfa48e94824444900b2efb1fb890ff (127.0.98.1:35871)
I20260927 21:09:30.520289 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36151
Sep 27 21:09:30 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:30.694928 13463 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:30.695111 13463 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:30.695389 13131 ts_manager.cc:194] Registered new tserver with Master: 9399fe230b594fa6860f55024b2bb12e (127.0.98.2:44107)
I20260927 21:09:30.695901 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39987
I20260927 21:09:30.865780 13597 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:30.865934 13597 heartbeater.cc:511] Master 127.0.98.62:40119 requested a full tablet report, sending...
I20260927 21:09:30.866225 13131 ts_manager.cc:194] Registered new tserver with Master: f22ba94e179b42d08bbcf8304c6e048f (127.0.98.3:46337)
I20260927 21:09:30.866758 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33259
I20260927 21:09:30.876416 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:09:30.887801 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.879391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.0.98.62:40119)
0927 21:09:30.879681 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:30.879683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:30.879795 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:09:30.879890 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:30.879891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:30.879904 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:30.879946 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:30.880684 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.881450 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.882533 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.882864 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.882916 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.884592 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:30.884594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:30.884595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:30.884603 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:30.886373 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:30.886795 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:30.886796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:30.886797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:30.886865 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:30.887142 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:30.887143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:30.887144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:30.887286 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:30.887362 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:30.887575 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:09:30.887682 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":66,"threads_started":1}
I20260927 21:09:30.890512 13131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:09:30.892795 13131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:09:30.910120 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.904876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46337 (local address 127.0.0.1:40764)
0927 21:09:30.905443 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:30.905484 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:30.905544 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:30.905931 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:30.905934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:30.905936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:30.906001 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:30.906002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.907296 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.907298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:30.908590 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:30.908593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.909792 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.909793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:30.909875 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.909891 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:09:30.909940 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:09:30.909989 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":67,"threads_started":1}
I20260927 21:09:30.910398 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.905112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.0.98.1:35871)
0927 21:09:30.906317 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:30.906322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:30.906330 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:09:30.906371 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:30.906373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:30.906388 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:30.906574 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:30.906639 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.907659 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.909103 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.909791 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.910164 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.910253 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:30.910281 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 21:09:30.910320 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":59,"threads_started":1}
I20260927 21:09:30.910125 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.904883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35871 (local address 127.0.0.1:38316)
0927 21:09:30.906095 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:30.906104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:30.906134 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:30.906481 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:30.906482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:30.906484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:30.906517 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:30.906519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.907756 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.907757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:30.909004 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:30.909006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.909890 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.909891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:30.909951 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.909956 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:09:30.909983 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 21:09:30.910021 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"thread_start_us":90,"threads_started":1}
I20260927 21:09:30.911991 13263 tablet_service.cc:1467] Processing CreateTablet for tablet 878ce92b62c144c8944f32d96ebf20c8 (DEFAULT_TABLE table=test-table [id=0955f886b9494c6bbc019bd9a1f96248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:30.912364 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878ce92b62c144c8944f32d96ebf20c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:30.914125 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.905295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44107 (local address 127.0.0.1:39668)
0927 21:09:30.906192 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:30.906199 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:30.906226 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:30.910693 (+ 4467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:30.910694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:30.910696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:30.910732 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:30.910734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.912016 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.912017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:30.913057 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:30.913060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.913937 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.913938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:30.913994 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.913998 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:30.914025 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 21:09:30.914059 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":64,"threads_started":1}
I20260927 21:09:30.914489 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.905406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.98.2:44107)
0927 21:09:30.910531 (+ 5125us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:30.910536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:30.910546 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:09:30.910578 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:30.910580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:30.910595 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:30.910793 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:30.910865 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.911920 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.913158 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.913842 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.914265 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.914362 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:30.914387 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 21:09:30.914423 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5091,"thread_start_us":56,"threads_started":1}
I20260927 21:09:30.915059 13397 tablet_service.cc:1467] Processing CreateTablet for tablet 878ce92b62c144c8944f32d96ebf20c8 (DEFAULT_TABLE table=test-table [id=0955f886b9494c6bbc019bd9a1f96248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:30.915388 13397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878ce92b62c144c8944f32d96ebf20c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:30.917248 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:30.905110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.98.3:46337)
0927 21:09:30.905663 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:30.905668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:30.905677 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:09:30.905711 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:30.905713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:30.905727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:30.905788 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:30.906065 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.907195 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.908853 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:30.909683 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:30.916999 (+ 7316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:30.917091 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:30.917119 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 21:09:30.917168 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":54,"threads_started":1}
I20260927 21:09:30.917982 13531 tablet_service.cc:1467] Processing CreateTablet for tablet 878ce92b62c144c8944f32d96ebf20c8 (DEFAULT_TABLE table=test-table [id=0955f886b9494c6bbc019bd9a1f96248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:30.918291 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878ce92b62c144c8944f32d96ebf20c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:30.918977 13632 tablet_bootstrap.cc:492] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: Bootstrap starting.
I20260927 21:09:30.919736 13632 tablet_bootstrap.cc:654] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:30.920032 13632 log.cc:824] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:30.920348 13633 tablet_bootstrap.cc:492] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Bootstrap starting.
I20260927 21:09:30.920940 13633 tablet_bootstrap.cc:654] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:30.921195 13633 log.cc:824] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:30.921916 13632 tablet_bootstrap.cc:492] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: No bootstrap required, opened a new log
I20260927 21:09:30.921980 13632 ts_tablet_manager.cc:1397] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:09:30.924418 13632 raft_consensus.cc:348] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:30.924568 13632 raft_consensus.cc:374] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:30.924595 13632 raft_consensus.cc:729] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdfa48e94824444900b2efb1fb890ff, State: Initialized, Role: FOLLOWER
I20260927 21:09:30.925024 13633 tablet_bootstrap.cc:492] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: No bootstrap required, opened a new log
I20260927 21:09:30.925092 13633 ts_tablet_manager.cc:1397] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:09:30.927493 13633 raft_consensus.cc:348] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:30.927660 13633 raft_consensus.cc:374] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:30.927690 13633 raft_consensus.cc:729] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9399fe230b594fa6860f55024b2bb12e, State: Initialized, Role: FOLLOWER
I20260927 21:09:30.927842 13632 consensus_queue.cc:260] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:30.928131 13632 ts_tablet_manager.cc:1428] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260927 21:09:30.928287 13329 heartbeater.cc:503] Master 127.0.98.62:40119 was elected leader, sending a full tablet report...
I20260927 21:09:30.928292 13633 consensus_queue.cc:260] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:30.928537 13633 ts_tablet_manager.cc:1428] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:09:30.929253 13463 heartbeater.cc:503] Master 127.0.98.62:40119 was elected leader, sending a full tablet report...
I20260927 21:09:30.930689 13636 tablet_bootstrap.cc:492] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Bootstrap starting.
I20260927 21:09:30.931416 13636 tablet_bootstrap.cc:654] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Neither blocks nor log segments found. Creating new log.
Sep 27 21:09:30 dist-test-slave-8bgs krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 21:09:30.931704 13636 log.cc:824] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Log is configured to *not* fsync() on all Append() calls
W20260927 21:09:30.931710 13464 tablet.cc:2369] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:30.934721 13636 tablet_bootstrap.cc:492] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: No bootstrap required, opened a new log
I20260927 21:09:30.934798 13636 ts_tablet_manager.cc:1397] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 21:09:30.937722 13636 raft_consensus.cc:348] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:30.937899 13636 raft_consensus.cc:374] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:30.937940 13636 raft_consensus.cc:729] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22ba94e179b42d08bbcf8304c6e048f, State: Initialized, Role: FOLLOWER
I20260927 21:09:30.938078 13636 consensus_queue.cc:260] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:30.938347 13636 ts_tablet_manager.cc:1428] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:09:30.938585 13597 heartbeater.cc:503] Master 127.0.98.62:40119 was elected leader, sending a full tablet report...
W20260927 21:09:31.000425 13330 tablet.cc:2369] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:31.007409 13640 raft_consensus.cc:482] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:31.007563 13640 raft_consensus.cc:504] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:31.007969 13640 leader_election.cc:288] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cdfa48e94824444900b2efb1fb890ff (127.0.98.1:35871), 9399fe230b594fa6860f55024b2bb12e (127.0.98.2:44107)
I20260927 21:09:31.012669 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:31.008306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36755 (local address 127.0.98.2:44107)
0927 21:09:31.008359 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:31.008362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:31.008717 (+ 355us) server_negotiation.cc:407] Connection header received
0927 21:09:31.008949 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:31.008950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:31.008965 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:31.008999 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:31.009243 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.010051 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.011676 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.012271 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.012320 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:31.012332 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:31.012556 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 21:09:31.012599 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:09:31.012967 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:31.008249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44107 (local address 127.0.98.3:36755)
0927 21:09:31.008644 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:31.008656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:31.008747 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:31.009074 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:31.009075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:31.009077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:31.009172 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:31.009174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.010183 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.010184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:31.011572 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:31.011575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.012402 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.012403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:31.012475 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:31.012479 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:31.012804 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 21:09:31.012849 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":97,"threads_started":1}
I20260927 21:09:31.013296 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878ce92b62c144c8944f32d96ebf20c8" candidate_uuid: "f22ba94e179b42d08bbcf8304c6e048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9399fe230b594fa6860f55024b2bb12e" is_pre_election: true
I20260927 21:09:31.013494 13400 raft_consensus.cc:2454] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22ba94e179b42d08bbcf8304c6e048f in term 0.
I20260927 21:09:31.013767 13484 leader_election.cc:302] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9399fe230b594fa6860f55024b2bb12e, f22ba94e179b42d08bbcf8304c6e048f; no voters:
I20260927 21:09:31.014320 13640 raft_consensus.cc:2781] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:09:31.014406 13640 raft_consensus.cc:482] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:31.014430 13640 raft_consensus.cc:3037] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:31.015301 13640 raft_consensus.cc:504] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:31.015451 13640 leader_election.cc:288] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [CANDIDATE]: Term 1 election: Requested vote from peers 6cdfa48e94824444900b2efb1fb890ff (127.0.98.1:35871), 9399fe230b594fa6860f55024b2bb12e (127.0.98.2:44107)
I20260927 21:09:31.015647 13400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878ce92b62c144c8944f32d96ebf20c8" candidate_uuid: "f22ba94e179b42d08bbcf8304c6e048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9399fe230b594fa6860f55024b2bb12e"
I20260927 21:09:31.015740 13400 raft_consensus.cc:3037] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:31.017024 13400 raft_consensus.cc:2454] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22ba94e179b42d08bbcf8304c6e048f in term 1.
I20260927 21:09:31.017241 13484 leader_election.cc:302] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9399fe230b594fa6860f55024b2bb12e, f22ba94e179b42d08bbcf8304c6e048f; no voters:
I20260927 21:09:31.017323 13640 raft_consensus.cc:2781] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:31.017452 13640 raft_consensus.cc:686] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 1 LEADER]: Becoming Leader. State: Replica: f22ba94e179b42d08bbcf8304c6e048f, State: Running, Role: LEADER
I20260927 21:09:31.017544 13640 consensus_queue.cc:237] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } }
I20260927 21:09:31.018395 13131 catalog_manager.cc:5495] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f reported cstate change: term changed from 0 to 1, leader changed from <none> to f22ba94e179b42d08bbcf8304c6e048f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f22ba94e179b42d08bbcf8304c6e048f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f22ba94e179b42d08bbcf8304c6e048f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46337 } health_report { overall_health: HEALTHY } } }
I20260927 21:09:31.021224 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:31.008667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58687 (local address 127.0.98.1:35871)
0927 21:09:31.008711 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:31.008713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:31.013254 (+ 4541us) server_negotiation.cc:407] Connection header received
0927 21:09:31.013369 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:31.013371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:31.013383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:31.013415 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:31.013591 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.014240 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.020225 (+ 5985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.020848 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.020898 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:31.020917 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:31.021109 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 21:09:31.021155 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:09:31.021425 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:31.008486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35871 (local address 127.0.98.3:58687)
0927 21:09:31.013200 (+ 4714us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:31.013211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:31.013280 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:31.013480 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:31.013481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:31.013482 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:31.013513 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:31.013514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.018161 (+ 4647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.018161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:31.020119 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:31.020122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.020975 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.020976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:31.021034 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:31.021037 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:09:31.021313 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 21:09:31.021363 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4702,"thread_start_us":81,"threads_started":1}
I20260927 21:09:31.021999 13266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878ce92b62c144c8944f32d96ebf20c8" candidate_uuid: "f22ba94e179b42d08bbcf8304c6e048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdfa48e94824444900b2efb1fb890ff" is_pre_election: true
I20260927 21:09:31.022158 13266 raft_consensus.cc:2454] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22ba94e179b42d08bbcf8304c6e048f in term 0.
I20260927 21:09:31.022290 13265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878ce92b62c144c8944f32d96ebf20c8" candidate_uuid: "f22ba94e179b42d08bbcf8304c6e048f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdfa48e94824444900b2efb1fb890ff"
I20260927 21:09:31.022344 13265 raft_consensus.cc:3037] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:31.023211 13265 raft_consensus.cc:2454] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22ba94e179b42d08bbcf8304c6e048f in term 1.
I20260927 21:09:31.039677 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:31.036207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.0.98.3:46337)
0927 21:09:31.036329 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:31.036331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:31.036340 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:09:31.036389 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:31.036390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:31.036418 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:31.036504 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:09:31.036680 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.037421 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.038532 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:31.038812 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:31.038872 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:31.039057 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:09:31.039237 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:09:31.039312 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:31.039548 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:09:31.039574 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 21:09:31.041884 13400 raft_consensus.cc:1261] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 1 FOLLOWER]: Refusing update from remote peer f22ba94e179b42d08bbcf8304c6e048f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:31.042301 13644 consensus_queue.cc:1035] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9399fe230b594fa6860f55024b2bb12e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:31.046559 13265 raft_consensus.cc:1261] T 878ce92b62c144c8944f32d96ebf20c8 P 6cdfa48e94824444900b2efb1fb890ff [term 1 FOLLOWER]: Refusing update from remote peer f22ba94e179b42d08bbcf8304c6e048f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:31.046947 13644 consensus_queue.cc:1035] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cdfa48e94824444900b2efb1fb890ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:31.051002 13647 mvcc.cc:205] Tried to move back new op lower bound from 7334065647784206336 to 7334065647688388608. Current Snapshot: MvccSnapshot[applied={T|T < 7334065647784206336}]
I20260927 21:09:31.051218 13650 mvcc.cc:205] Tried to move back new op lower bound from 7334065647784206336 to 7334065647688388608. Current Snapshot: MvccSnapshot[applied={T|T < 7334065647784206336}]
I20260927 21:09:31.055694 13131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:31.055788 13131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:31.056552 13131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07baf5ec6c334e289390d45a47e3b5c6: Sending DeleteTablet for 3 replicas of tablet 878ce92b62c144c8944f32d96ebf20c8
I20260927 21:09:31.056874 13531 tablet_service.cc:1514] Processing DeleteTablet for tablet 878ce92b62c144c8944f32d96ebf20c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40764
I20260927 21:09:31.057161 13263 tablet_service.cc:1514] Processing DeleteTablet for tablet 878ce92b62c144c8944f32d96ebf20c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38316
I20260927 21:09:31.057513 13397 tablet_service.cc:1514] Processing DeleteTablet for tablet 878ce92b62c144c8944f32d96ebf20c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260927 21:09:31.058081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13169
I20260927 21:09:31.060433 13648 mvcc.cc:205] Tried to move back new op lower bound from 7334065647784206336 to 7334065647688388608. Current Snapshot: MvccSnapshot[applied={T|T < 7334065647784206336}]
I20260927 21:09:31.060616 13655 tablet_replica.cc:330] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: stopping tablet replica
I20260927 21:09:31.064445 13657 tablet_replica.cc:330] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: stopping tablet replica
I20260927 21:09:31.064594 13655 raft_consensus.cc:2229] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:09:31.064749 13657 raft_consensus.cc:2229] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:31.064922 13655 raft_consensus.cc:2258] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:31.065037 13657 raft_consensus.cc:2258] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:31.065603 13655 ts_tablet_manager.cc:1905] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:31.071220 13655 ts_tablet_manager.cc:1918] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:31.071306 13655 log.cc:1196] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/878ce92b62c144c8944f32d96ebf20c8
I20260927 21:09:31.071580 13655 ts_tablet_manager.cc:1939] T 878ce92b62c144c8944f32d96ebf20c8 P f22ba94e179b42d08bbcf8304c6e048f: Deleting consensus metadata
I20260927 21:09:31.072096 13118 catalog_manager.cc:4841] TS f22ba94e179b42d08bbcf8304c6e048f (127.0.98.3:46337): tablet 878ce92b62c144c8944f32d96ebf20c8 (table test-table [id=0955f886b9494c6bbc019bd9a1f96248]) successfully deleted
I20260927 21:09:31.072491 13657 ts_tablet_manager.cc:1905] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:31.074115 13657 ts_tablet_manager.cc:1918] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:31.074250 13657 log.cc:1196] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/878ce92b62c144c8944f32d96ebf20c8
I20260927 21:09:31.074564 13657 ts_tablet_manager.cc:1939] T 878ce92b62c144c8944f32d96ebf20c8 P 9399fe230b594fa6860f55024b2bb12e: Deleting consensus metadata
I20260927 21:09:31.075019 13115 catalog_manager.cc:4841] TS 9399fe230b594fa6860f55024b2bb12e (127.0.98.2:44107): tablet 878ce92b62c144c8944f32d96ebf20c8 (table test-table [id=0955f886b9494c6bbc019bd9a1f96248]) successfully deleted
W20260927 21:09:31.075309 13117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35871 (error 108)
I20260927 21:09:31.083232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13333
I20260927 21:09:31.084856 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:31.084698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35871 (local address 127.0.0.1:38318)
0927 21:09:31.084777 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:31.084818 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 21:09:31.085110 13117 catalog_manager.cc:4584] TS 6cdfa48e94824444900b2efb1fb890ff (127.0.98.1:35871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 878ce92b62c144c8944f32d96ebf20c8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35871: connect: Connection refused (error 111)
I20260927 21:09:31.089011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13466
I20260927 21:09:31.095117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13098
I20260927 21:09:31.099215 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:09:31.443847 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:09:31.443935 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:09:31.788303 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:09:31.788 UTC [12165] LOG: received smart shutdown request
2026-09-27 21:09:31.792 UTC [12165] LOG: background worker "logical replication launcher" (PID 12174) exited with exit code 1
2026-09-27 21:09:31.792 UTC [12169] LOG: shutting down
2026-09-27 21:09:31.947 UTC [12165] LOG: database system is shut down
2026-09-27T21:09:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47803 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:09:32 dist-test-slave-8bgs krb5kdc[13668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:09:32 dist-test-slave-8bgs krb5kdc[13668](info): set up 2 sockets
Sep 27 21:09:32 dist-test-slave-8bgs krb5kdc[13668](info): commencing operation
krb5kdc: starting...
W20260927 21:09:33.074908 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:09:33.187582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33371,127.0.98.61:39301,127.0.98.60:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:33.266566 13681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:33.266814 13681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:33.266855 13681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:33.268064 13681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:33.268131 13681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:33.268155 13681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:33.268186 13681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:33.268211 13681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:33.269690 13681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33371,127.0.98.61:39301,127.0.98.60:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:33.269968 13681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:33.270187 13681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:33.275535 13687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:33.275525 13686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:33.275530 13689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:33.276496 13681 server_base.cc:1034] running on GCE node
I20260927 21:09:33.276777 13681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:33.277104 13681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:33.278245 13681 hybrid_clock.cc:648] HybridClock initialized: now 1790543373278222 us; error 36 us; skew 500 ppm
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:33.287781 13681 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:33.288259 13681 webserver.cc:460] Webserver started at http://127.0.98.62:43131/ using document root <none> and password file <none>
I20260927 21:09:33.288483 13681 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:33.288552 13681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:33.288702 13681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:33.289789 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "36f0390d07c348149360d285386ede7b"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "223148a763dfd4765118435fcaa180b6"
server_key_iv: "da4847e9687ec98962364fb3708a5e75"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.290162 13681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "36f0390d07c348149360d285386ede7b"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "223148a763dfd4765118435fcaa180b6"
server_key_iv: "da4847e9687ec98962364fb3708a5e75"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.291610 13681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:09:33.292443 13696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:33.292611 13681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:09:33.292739 13681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "36f0390d07c348149360d285386ede7b"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "223148a763dfd4765118435fcaa180b6"
server_key_iv: "da4847e9687ec98962364fb3708a5e75"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.292816 13681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:33.300038 13681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:33.302259 13681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:33.302412 13681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:33.306313 13681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33371
I20260927 21:09:33.306342 13748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33371 every 8 connection(s)
I20260927 21:09:33.306685 13681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:33.307528 13749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:33.309046 13749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:09:33.311898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13681
I20260927 21:09:33.312038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:09:33.312222 392 external_mini_cluster.cc:1428] Setting key 081b628d49f5fe5c7b326975e08baa9c
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:33.318693 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.309675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33371 (local address 127.0.0.1:49686)
0927 21:09:33.309814 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.309847 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.310259 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.310499 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.310502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.310739 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.310889 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.310893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.311940 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.311941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.312758 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.312761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.313269 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.313270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.313332 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.313864 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.313873 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.315481 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.317238 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.317244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.317246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.317436 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.317682 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.317682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.317683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.317772 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.318116 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.318121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.318181 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.318292 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.318521 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":92,"thread_start_us":63,"threads_started":1}
I20260927 21:09:33.318698 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.309820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.0.98.62:33371)
0927 21:09:33.309955 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.309964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.309991 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:09:33.310383 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.310385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.310400 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.310477 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.310967 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.311830 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.312858 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.313167 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.313299 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.315585 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.315591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.315591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.315598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.317141 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.317532 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.317535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.317535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.317613 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.317839 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.317840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.317840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.318043 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.318139 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.318329 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.318460 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 21:09:33.319480 13749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:09:33.319828 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.319704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59088)
0927 21:09:33.319753 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.319798 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:09:33.319958 13700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
I20260927 21:09:33.320333 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.320240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59100)
0927 21:09:33.320289 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.320313 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:09:33.320438 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39301: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 21:09:33.333969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39301
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33371,127.0.98.61:39301,127.0.98.60:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:33.369642 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } attempt: 1
I20260927 21:09:33.370113 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.369980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59110)
0927 21:09:33.370043 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.370085 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:09:33.370512 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.370377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59122)
0927 21:09:33.370438 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.370480 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 21:09:33.370649 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39301: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
W20260927 21:09:33.413705 13756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:33.413910 13756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:33.413932 13756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:33.415123 13756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:33.415164 13756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:33.415179 13756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:33.415191 13756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:33.415230 13756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:33.416721 13756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33371,127.0.98.61:39301,127.0.98.60:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:33.416971 13756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:33.417192 13756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:33.421896 13761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:33.421906 13762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:33.421911 13764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:33.422174 13756 server_base.cc:1034] running on GCE node
I20260927 21:09:33.422425 13756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:33.422736 13756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:33.423907 13756 hybrid_clock.cc:648] HybridClock initialized: now 1790543373423882 us; error 48 us; skew 500 ppm
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:33.426316 13756 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 21:09:33.426739 13756 webserver.cc:460] Webserver started at http://127.0.98.61:34737/ using document root <none> and password file <none>
I20260927 21:09:33.426975 13756 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:33.427045 13756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:33.427218 13756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:33.428335 13756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data/instance:
uuid: "e392f76d146f4176b94bc524fd8a7828"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "a3a8080f4e6a1c7f66e1bf7554ce968f"
server_key_iv: "3003765b9dbbe7ac319cc195e805d19a"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.428681 13756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal/instance:
uuid: "e392f76d146f4176b94bc524fd8a7828"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "a3a8080f4e6a1c7f66e1bf7554ce968f"
server_key_iv: "3003765b9dbbe7ac319cc195e805d19a"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.429944 13756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:09:33.430610 13771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:33.430789 13756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:33.430892 13756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal
uuid: "e392f76d146f4176b94bc524fd8a7828"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "a3a8080f4e6a1c7f66e1bf7554ce968f"
server_key_iv: "3003765b9dbbe7ac319cc195e805d19a"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.430965 13756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:33.438815 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } attempt: 2
I20260927 21:09:33.439327 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.439199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59138)
0927 21:09:33.439256 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.439297 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:09:33.439759 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.439631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59154)
0927 21:09:33.439703 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.439735 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:09:33.439885 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39301: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39301: connect: Connection refused (error 111)
I20260927 21:09:33.452139 13756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:33.454479 13756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:33.454591 13756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:33.458640 13756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39301
I20260927 21:09:33.458670 13823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39301 every 8 connection(s)
I20260927 21:09:33.459033 13756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:09:33.459928 13824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:33.461419 13824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:33.468592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13756
I20260927 21:09:33.468709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-1/wal/instance
I20260927 21:09:33.468909 392 external_mini_cluster.cc:1428] Setting key 89822225644036554ccb955f7ee4bca5
I20260927 21:09:33.470010 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.462079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.0.98.62:33371)
0927 21:09:33.462141 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.462143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.462363 (+ 220us) server_negotiation.cc:407] Connection header received
0927 21:09:33.462797 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.462798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.462809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.462872 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.463506 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.463971 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.464899 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.465083 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.465134 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.467016 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.467018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.467018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.467024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.468495 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.469041 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.469042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.469042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.469104 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.469362 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.469363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.469364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.469479 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.469550 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.469835 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.469920 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:09:33.470202 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.462069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33371 (local address 127.0.0.1:49696)
0927 21:09:33.462228 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.462278 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.462714 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.462926 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.462930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.463238 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.463429 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.463433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.464047 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.464047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.464825 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.464827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.465152 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.465153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.465265 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.465667 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.465675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.466935 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.468604 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.468612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.468614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.468927 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.469172 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.469173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.469174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.469288 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.469555 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.469562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.469697 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.469803 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.469959 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260927 21:09:33.470803 13824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 21:09:33.478849 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.471062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59168)
0927 21:09:33.471132 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.471143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.471183 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.471464 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.471464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.471629 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.471687 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.471689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.473026 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.473027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.473513 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.473515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.474223 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.474223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.474277 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.474697 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.474701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.475814 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.477787 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.477788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.477789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.477935 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.478156 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.478156 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.478157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.478203 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.478521 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.478521 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.478560 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.478635 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.478736 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:09:33.478927 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.471137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.0.98.61:39301)
0927 21:09:33.471304 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.471309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.471334 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:09:33.471356 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.471358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.471372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.471445 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.471775 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.472944 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.473598 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.474135 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.474220 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.475897 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.475906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.475906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.475913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.477720 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.478027 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.478030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.478030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.478092 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.478241 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.478241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.478242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.478448 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.478542 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.478667 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.478788 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260927 21:09:33.479775 13824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 21:09:33.480154 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.480021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53022)
0927 21:09:33.480071 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.480124 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:09:33.480331 13773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
I20260927 21:09:33.480756 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.480649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53034)
0927 21:09:33.480696 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.480725 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:09:33.480865 13824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 21:09:33.494676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41239
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33371,127.0.98.61:39301,127.0.98.60:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:33.530097 13824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } attempt: 1
I20260927 21:09:33.530594 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.530453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53044)
0927 21:09:33.530525 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.530564 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 21:09:33.530939 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.530844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53054)
0927 21:09:33.530890 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.530913 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:09:33.531042 13824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
I20260927 21:09:33.531037 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } attempt: 3
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:09:33.538582 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.531402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.98.61:39301)
0927 21:09:33.531438 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.531440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.531452 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:09:33.531524 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.531525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.531535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.531587 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.531949 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.532443 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.533577 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.533794 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.533850 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.536020 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.536022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.536022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.536030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.537444 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.537805 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.537807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.537807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.537865 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.538063 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.538064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.538064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.538181 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.538256 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.538408 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.538487 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:09:33.538584 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.531354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59170)
0927 21:09:33.531410 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.531421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.531464 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.531615 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.531616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.531798 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.531875 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.531877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.532532 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.532534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.533474 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.533478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.533874 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.533875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.533959 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.534477 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.534483 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.535897 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.537546 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.537547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.537548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.537732 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.537932 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.537933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.537933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.538000 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.538247 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.538248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.538297 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.538384 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.538487 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:09:33.538923 13749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:09:33.539287 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.539187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53066)
0927 21:09:33.539236 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.539263 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:09:33.539690 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.539590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53078)
0927 21:09:33.539650 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.539673 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:09:33.539793 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
I20260927 21:09:33.570926 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } attempt: 1
I20260927 21:09:33.571444 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.571312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53094)
0927 21:09:33.571375 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.571417 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:09:33.571838 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.571755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53108)
0927 21:09:33.571799 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.571824 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:09:33.571942 13749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
W20260927 21:09:33.579661 13831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:33.579847 13831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:33.579869 13831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:33.580999 13831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:33.581038 13831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:33.581053 13831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:33.581065 13831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:33.581118 13831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:33.582569 13831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33371,127.0.98.61:39301,127.0.98.60:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:33.582828 13831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:33.583051 13831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:33.587661 13839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:33.587757 13831 server_base.cc:1034] running on GCE node
W20260927 21:09:33.587658 13837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:33.587823 13836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:33.588073 13831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:33.588316 13831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:33.589497 13831 hybrid_clock.cc:648] HybridClock initialized: now 1790543373589484 us; error 41 us; skew 500 ppm
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:33.591914 13831 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 21:09:33.592383 13831 webserver.cc:460] Webserver started at http://127.0.98.60:35909/ using document root <none> and password file <none>
I20260927 21:09:33.592607 13831 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:33.592674 13831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:33.592823 13831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:33.593848 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data/instance:
uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "a3720ec64a1f740b24d8555a482eb945"
server_key_iv: "d5d47032528f236e7b670b2849f51b33"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.594209 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal/instance:
uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "a3720ec64a1f740b24d8555a482eb945"
server_key_iv: "d5d47032528f236e7b670b2849f51b33"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.595651 13831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:09:33.596347 13846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:33.596510 13831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:09:33.596606 13831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal
uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c"
format_stamp: "Formatted at 2026-09-27 21:09:33 on dist-test-slave-8bgs"
server_key: "a3720ec64a1f740b24d8555a482eb945"
server_key_iv: "d5d47032528f236e7b670b2849f51b33"
server_key_version: "encryptionkey@0"
I20260927 21:09:33.596696 13831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:33.599169 13824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } attempt: 2
I20260927 21:09:33.599555 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.599410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53112)
0927 21:09:33.599474 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.599517 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:09:33.599939 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.599845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53124)
0927 21:09:33.599891 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.599918 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:09:33.600050 13824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41239: connect: Connection refused (error 111)
I20260927 21:09:33.607615 13831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:33.609344 13831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:33.609500 13831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:33.613556 13831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41239
I20260927 21:09:33.613584 13898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41239 every 8 connection(s)
I20260927 21:09:33.613924 13831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:09:33.614728 13899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:33.616289 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:09:33.620596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13831
I20260927 21:09:33.620745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790543220945705-392-0/minicluster-data/master-2/wal/instance
I20260927 21:09:33.620934 392 external_mini_cluster.cc:1428] Setting key 895824ec60355e210ef27f706204936f
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:33.625788 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.616977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.0.98.62:33371)
0927 21:09:33.617033 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.617035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.617198 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:09:33.617603 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.617604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.617612 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.617669 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.618243 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.618730 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.619730 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.619904 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.619959 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.622523 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.622524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.622524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.622531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.624189 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.624642 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.624643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.624643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.624697 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.624954 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.624955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.624956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.625104 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.625179 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.625570 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.625659 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:09:33.625859 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.616927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33371 (local address 127.0.0.1:49708)
0927 21:09:33.617076 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.617114 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.617528 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.617755 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.617759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.617991 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.618173 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.618176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.618805 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.618806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.619630 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.619632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.619991 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.619992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.620102 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.620607 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.620615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.622429 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.624273 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.624279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.624280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.624566 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.624766 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.624766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.624767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.624889 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.625247 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.625254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.625421 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.625535 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.625682 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260927 21:09:33.626425 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:09:33.630539 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.621290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.0.98.60:41239)
0927 21:09:33.621422 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.621427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.621542 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:09:33.621595 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.621597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.621613 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.621671 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.622332 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.623416 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.624442 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.624927 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.624989 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.627691 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.627696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.627697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.627702 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.629360 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.629753 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.629755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.629756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.629808 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.629989 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.629990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.629990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.630181 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.630306 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.630415 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.630470 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:09:33.633656 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.626775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.0.98.61:39301)
0927 21:09:33.626852 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.626855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.626865 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:09:33.626890 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.626891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.626902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.626965 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.627346 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.628021 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.628977 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.629151 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.629205 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.631170 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.631171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.631171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.631178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.632596 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.632926 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.632927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.632927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.632983 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.633178 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.633179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.633179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.633286 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.633355 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.633502 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.633570 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:09:33.633656 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.626697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39301 (local address 127.0.0.1:59174)
0927 21:09:33.626747 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.626759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.626803 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.626975 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.626976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.627149 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.627225 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.627227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.628095 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.628096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.628894 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.628897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.629223 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.629224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.629304 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.629666 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.629670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.631052 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.632676 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.632677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.632677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.632839 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.633068 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.633069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.633069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.633131 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.633359 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.633360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.633419 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.633480 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.633562 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:09:33.634265 13899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:09:33.639397 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.631627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.0.98.62:33371)
0927 21:09:33.631682 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.631684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.631691 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:09:33.631710 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.631711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.631719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.631771 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.632368 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.632781 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.633457 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.633659 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.633841 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.636094 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.636095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.636096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.636102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.637706 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.638073 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.638074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.638074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.638120 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.638323 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.638324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.638324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.638454 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.639173 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.639222 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.639283 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:09:33.641392 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.634500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53128)
0927 21:09:33.634536 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.634546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.634586 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.634705 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.634706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.634828 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.634849 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.634850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.635757 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.635758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.636349 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.636351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.636770 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.636772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.636833 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.637111 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.637116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.638469 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.640337 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.640337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.640338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.640489 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.640681 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.640681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.640682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.640851 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.641069 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.641070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.641111 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.641186 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.641293 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 21:09:33.641474 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.634500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.98.60:41239)
0927 21:09:33.634539 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.634541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.634573 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:09:33.634639 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.634640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.634649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.634702 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.634920 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.635659 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.636424 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.636699 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.636749 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.638545 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.638546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.638546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.638553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.640258 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.640558 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.640559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.640560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.640618 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.640914 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.640915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.640915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.641006 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.641055 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.641202 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.641333 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":27}
I20260927 21:09:33.642724 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: Bootstrap starting.
I20260927 21:09:33.643620 13899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:33.643899 13899 log.cc:824] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:09:33.644661 13899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: No bootstrap required, opened a new log
I20260927 21:09:33.646569 13899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.646726 13899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:33.646759 13899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b6ab0f339349ef8c97fd97ae0cdd6c, State: Initialized, Role: FOLLOWER
I20260927 21:09:33.646924 13899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.647068 13749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } attempt: 2
I20260927 21:09:33.647233 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.640082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.0.98.61:39301)
0927 21:09:33.640133 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.640135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.640142 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:09:33.640163 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.640163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.640179 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.640227 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.640866 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.641537 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.642221 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.642422 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.642503 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.644759 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.644761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.644761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.644768 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.646067 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.646438 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.646439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.646439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.646484 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.646676 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.646676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.646676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.646825 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.646995 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.647031 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.647128 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:09:33.647439 13899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:33.647503 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.647636 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:09:33.648893 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 21:09:33.649379 13916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:09:33.650159 13916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:09:33.650532 13831 master_runner.cc:186] Error getting master registration for 127.0.98.62:33371: OK, instance_id { permanent_uuid: "36f0390d07c348149360d285386ede7b" instance_seqno: 1790543373305266 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:09:33.653059 13831 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:09:33.657636 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.647332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53140)
0927 21:09:33.647414 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.647424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.647463 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.647693 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.647695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.647858 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.647899 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.647901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.648785 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.648786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.649541 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.649544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.649997 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.649998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.650058 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.653153 (+ 3095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.653159 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.654601 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.656467 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.656468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.656468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.656688 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.656904 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.656904 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.656905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.656970 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.657320 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.657321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.657370 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.657438 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.657529 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:09:33.658178 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.647537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.98.60:41239)
0927 21:09:33.647573 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.647575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.647581 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:09:33.647599 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.647600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.647609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.647653 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.647991 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.648693 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.649636 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.649926 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.649995 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.654686 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.654688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.654688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.654694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.656383 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.656776 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.656777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.656778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.656832 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.657038 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.657039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.657039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.657217 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.657291 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.658003 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.658079 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:09:33.659766 13749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: Bootstrap starting.
I20260927 21:09:33.660291 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.650537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.98.62:33371)
0927 21:09:33.650572 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.650574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.650837 (+ 263us) server_negotiation.cc:407] Connection header received
0927 21:09:33.650930 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.650931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.650940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.650968 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.651725 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.652407 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.653541 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.653834 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.653877 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.656118 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.656120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.656120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.656126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.658637 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.659023 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.659024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.659025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.659080 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.659712 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.659713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.659713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.659855 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.659928 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.660137 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.660211 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:09:33.660573 13749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:33.660643 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.650851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.0.98.60:41239)
0927 21:09:33.651276 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.651278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.651438 (+ 160us) server_negotiation.cc:407] Connection header received
0927 21:09:33.651459 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.651460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.651480 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.651529 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.652230 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.652997 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.654212 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.654528 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.654592 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.656267 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.656268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.656268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.656275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.658024 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.659228 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.659230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.659230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.659285 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.659500 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.659501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.659501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.659615 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.660095 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.660444 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.660527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":64,"threads_started":1}
I20260927 21:09:33.660869 13749 log.cc:824] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:33.662206 13749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: No bootstrap required, opened a new log
I20260927 21:09:33.663981 13749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.664131 13749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:33.664171 13749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f0390d07c348149360d285386ede7b, State: Initialized, Role: FOLLOWER
I20260927 21:09:33.664307 13749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.664604 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.664657 13749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:33.664754 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:09:33.667122 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.654146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.0.98.61:39301)
0927 21:09:33.654183 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.654185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.655245 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 21:09:33.656560 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.656562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.656573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.656605 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.660589 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.661312 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.662270 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.662539 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.662625 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.663991 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.663993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.663993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.664001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.665714 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.666074 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.666075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.666075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.666141 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.666340 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.666341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.666341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.666469 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.666550 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.666919 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.666991 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 21:09:33.667616 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:09:33.667689 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:09:33.667569 13681 master_runner.cc:186] Error getting master registration for 127.0.98.61:39301: OK, instance_id { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" instance_seqno: 1790543373457419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:09:33.669160 13681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:09:33.691263 13824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } attempt: 3
Sep 27 21:09:33 dist-test-slave-8bgs krb5kdc[13668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:09:33.698380 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.691609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.0.98.60:41239)
0927 21:09:33.691647 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:33.691648 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:33.691673 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:09:33.691742 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:33.691743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:33.691759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:33.691824 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:33.692102 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.692643 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.693591 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.693823 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.693900 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.695839 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:33.695840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:33.695841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:33.695847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:33.697247 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.697604 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.697605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.697606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.697661 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:33.697872 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:33.697873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:33.697873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:33.697981 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:33.698056 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:33.698225 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:09:33.698298 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:09:33.698380 13826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:33.691575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41239 (local address 127.0.0.1:53164)
0927 21:09:33.691625 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:33.691636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:33.691689 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:33.691828 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:33.691828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:33.691989 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:33.692028 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.692029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.692745 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.692746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.693510 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:33.693512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:33.693896 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:33.693897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:33.693951 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:33.694409 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:33.694416 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:33.695745 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:33.697347 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.697348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.697348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.697519 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.697743 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:33.697744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:33.697744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:33.697800 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:33.698047 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:33.698048 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:33.698091 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:09:33.698175 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:09:33.698282 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 21:09:33.700275 13824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: Bootstrap starting.
I20260927 21:09:33.701009 13824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:33.701241 13824 log.cc:824] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:33.701929 13824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: No bootstrap required, opened a new log
I20260927 21:09:33.703399 13824 raft_consensus.cc:348] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.703534 13824 raft_consensus.cc:374] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:33.703598 13824 raft_consensus.cc:729] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e392f76d146f4176b94bc524fd8a7828, State: Initialized, Role: FOLLOWER
I20260927 21:09:33.703742 13824 consensus_queue.cc:260] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.704028 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.704095 13824 sys_catalog.cc:564] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:33.704155 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:09:33.705147 13777 tablet.cc:2369] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:09:33.705879 13952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:09:33.705996 13952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:09:33.707957 13756 master_runner.cc:263] No leader master found from master e392f76d146f4176b94bc524fd8a7828
I20260927 21:09:33.708000 13756 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:09:33.774838 13941 raft_consensus.cc:482] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:33.774958 13941 raft_consensus.cc:504] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.775218 13941 leader_election.cc:288] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36f0390d07c348149360d285386ede7b (127.0.98.62:33371), 25b6ab0f339349ef8c97fd97ae0cdd6c (127.0.98.60:41239)
I20260927 21:09:33.775782 13723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e392f76d146f4176b94bc524fd8a7828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f0390d07c348149360d285386ede7b" is_pre_election: true
I20260927 21:09:33.775940 13723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e392f76d146f4176b94bc524fd8a7828 in term 0.
I20260927 21:09:33.776129 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e392f76d146f4176b94bc524fd8a7828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" is_pre_election: true
I20260927 21:09:33.776283 13873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e392f76d146f4176b94bc524fd8a7828 in term 0.
I20260927 21:09:33.776695 13773 leader_election.cc:302] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b6ab0f339349ef8c97fd97ae0cdd6c, e392f76d146f4176b94bc524fd8a7828; no voters:
I20260927 21:09:33.776911 13941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:09:33.776968 13941 raft_consensus.cc:482] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:33.776991 13941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:33.777927 13941 raft_consensus.cc:504] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.778057 13941 leader_election.cc:288] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [CANDIDATE]: Term 1 election: Requested vote from peers 36f0390d07c348149360d285386ede7b (127.0.98.62:33371), 25b6ab0f339349ef8c97fd97ae0cdd6c (127.0.98.60:41239)
I20260927 21:09:33.778532 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e392f76d146f4176b94bc524fd8a7828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c"
I20260927 21:09:33.778606 13873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:33.779171 13723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e392f76d146f4176b94bc524fd8a7828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f0390d07c348149360d285386ede7b"
I20260927 21:09:33.779268 13723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:33.780112 13723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e392f76d146f4176b94bc524fd8a7828 in term 1.
I20260927 21:09:33.780339 13773 leader_election.cc:302] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f0390d07c348149360d285386ede7b, e392f76d146f4176b94bc524fd8a7828; no voters:
I20260927 21:09:33.780891 13873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e392f76d146f4176b94bc524fd8a7828 in term 1.
I20260927 21:09:33.781129 13941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:33.781312 13941 raft_consensus.cc:686] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [term 1 LEADER]: Becoming Leader. State: Replica: e392f76d146f4176b94bc524fd8a7828, State: Running, Role: LEADER
I20260927 21:09:33.781414 13941 consensus_queue.cc:237] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } }
I20260927 21:09:33.781775 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e392f76d146f4176b94bc524fd8a7828. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.781855 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:33.782158 13956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:33.782428 13956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:33.784515 13873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [term 1 FOLLOWER]: Refusing update from remote peer e392f76d146f4176b94bc524fd8a7828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:33.784783 13723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [term 1 FOLLOWER]: Refusing update from remote peer e392f76d146f4176b94bc524fd8a7828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:33.785269 13941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:33.785398 13941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:33.787750 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e392f76d146f4176b94bc524fd8a7828. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.787835 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:09:33.789012 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e392f76d146f4176b94bc524fd8a7828. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.789116 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:09:33.790792 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.790876 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:33.791298 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.791375 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:09:33.791612 13957 mvcc.cc:205] Tried to move back new op lower bound from 7334065659018665984 to 7334065659009138688. Current Snapshot: MvccSnapshot[applied={T|T < 7334065659018665984}]
I20260927 21:09:33.792431 13960 mvcc.cc:205] Tried to move back new op lower bound from 7334065659018665984 to 7334065659009138688. Current Snapshot: MvccSnapshot[applied={T|T < 7334065659018665984}]
I20260927 21:09:33.792860 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.792940 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:33.793432 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e392f76d146f4176b94bc524fd8a7828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f0390d07c348149360d285386ede7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33371 } } peers { permanent_uuid: "e392f76d146f4176b94bc524fd8a7828" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39301 } } peers { permanent_uuid: "25b6ab0f339349ef8c97fd97ae0cdd6c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41239 } } }
I20260927 21:09:33.793529 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:09:33.793895 13958 mvcc.cc:205] Tried to move back new op lower bound from 7334065659018665984 to 7334065659009138688. Current Snapshot: MvccSnapshot[applied={T|T < 7334065659018665984}]
I20260927 21:09:33.794867 13956 catalog_manager.cc:1347] Generated new cluster ID: ab3d6dc9cf8540adafb2fe54917eac1c
I20260927 21:09:33.794915 13956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:09:33.824156 13956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:09:33.824967 13956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:33.834630 13956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e392f76d146f4176b94bc524fd8a7828: Generated new TSK 0
I20260927 21:09:33.834887 13956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:09:34.650977 13916 catalog_manager.cc:1259] Loaded cluster ID: ab3d6dc9cf8540adafb2fe54917eac1c
I20260927 21:09:34.651059 13916 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: loading cluster ID for follower catalog manager: success
I20260927 21:09:34.651930 13916 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: acquiring CA information for follower catalog manager: success
I20260927 21:09:34.652246 13916 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25b6ab0f339349ef8c97fd97ae0cdd6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:09:34.668134 13938 catalog_manager.cc:1259] Loaded cluster ID: ab3d6dc9cf8540adafb2fe54917eac1c
I20260927 21:09:34.668201 13938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: loading cluster ID for follower catalog manager: success
I20260927 21:09:34.668987 13938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: acquiring CA information for follower catalog manager: success
I20260927 21:09:34.669245 13938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36f0390d07c348149360d285386ede7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:09:34.884946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13681
I20260927 21:09:34.889988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13756
I20260927 21:09:34.894652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13831
2026-09-27T21:09:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2839 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:09:34 dist-test-slave-8bgs krb5kdc[13972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:09:34 dist-test-slave-8bgs krb5kdc[13972](info): set up 2 sockets
Sep 27 21:09:34 dist-test-slave-8bgs krb5kdc[13972](info): commencing operation
krb5kdc: starting...
W20260927 21:09:35.915272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:09:35 dist-test-slave-8bgs krb5kdc[13972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:09:36.029805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:36.109498 13985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:36.109723 13985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:36.109757 13985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:36.110895 13985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:36.110960 13985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:36.110983 13985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:36.111004 13985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:36.111033 13985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:36.112519 13985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:36.112777 13985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:36.112993 13985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:09:36.117671 13985 server_base.cc:1034] running on GCE node
W20260927 21:09:36.117709 13990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:36.117654 13993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:36.117645 13991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:36.118139 13985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:36.118387 13985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:36.119567 13985 hybrid_clock.cc:648] HybridClock initialized: now 1790543376119551 us; error 29 us; skew 500 ppm
Sep 27 21:09:36 dist-test-slave-8bgs krb5kdc[13972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:36.122013 13985 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:36.122454 13985 webserver.cc:460] Webserver started at http://127.0.98.62:32789/ using document root <none> and password file <none>
I20260927 21:09:36.122666 13985 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:36.122707 13985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:36.122809 13985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:36.123858 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "53dbb0fc95094c5998f753e068fc1810"
format_stamp: "Formatted at 2026-09-27 21:09:36 on dist-test-slave-8bgs"
server_key: "4d5ce191364a9c54d2fd68975069f9b1"
server_key_iv: "b1dfd6a3fe05e7bd064305bc96d67106"
server_key_version: "encryptionkey@0"
I20260927 21:09:36.124152 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "53dbb0fc95094c5998f753e068fc1810"
format_stamp: "Formatted at 2026-09-27 21:09:36 on dist-test-slave-8bgs"
server_key: "4d5ce191364a9c54d2fd68975069f9b1"
server_key_iv: "b1dfd6a3fe05e7bd064305bc96d67106"
server_key_version: "encryptionkey@0"
I20260927 21:09:36.125379 13985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:09:36.126046 14000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:36.126237 13985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:09:36.126322 13985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "53dbb0fc95094c5998f753e068fc1810"
format_stamp: "Formatted at 2026-09-27 21:09:36 on dist-test-slave-8bgs"
server_key: "4d5ce191364a9c54d2fd68975069f9b1"
server_key_iv: "b1dfd6a3fe05e7bd064305bc96d67106"
server_key_version: "encryptionkey@0"
I20260927 21:09:36.126374 13985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:36.136978 13985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:36.138671 13985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:36.138780 13985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:36.142318 13985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37463
I20260927 21:09:36.142378 14052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37463 every 8 connection(s)
I20260927 21:09:36.142693 13985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:36.143577 14053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:36.144129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 13985
I20260927 21:09:36.144222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:09:36.144438 392 external_mini_cluster.cc:1428] Setting key 6776cbbb1c60b67ef8d742bd7a43d39b
I20260927 21:09:36.146508 14053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810: Bootstrap starting.
I20260927 21:09:36.147233 14053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:36.147527 14053 log.cc:824] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:36.148186 14053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810: No bootstrap required, opened a new log
I20260927 21:09:36.150427 14053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dbb0fc95094c5998f753e068fc1810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37463 } }
I20260927 21:09:36.150550 14053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:36.150573 14053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dbb0fc95094c5998f753e068fc1810, State: Initialized, Role: FOLLOWER
I20260927 21:09:36.150704 14053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dbb0fc95094c5998f753e068fc1810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37463 } }
I20260927 21:09:36.150802 14053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:36.150838 14053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:36.150887 14053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 21:09:36 dist-test-slave-8bgs krb5kdc[13972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:36.151543 14053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dbb0fc95094c5998f753e068fc1810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37463 } }
I20260927 21:09:36.151652 14053 leader_election.cc:302] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53dbb0fc95094c5998f753e068fc1810; no voters:
I20260927 21:09:36.151815 14053 leader_election.cc:288] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:09:36.151866 14058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:36.152011 14058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [term 1 LEADER]: Becoming Leader. State: Replica: 53dbb0fc95094c5998f753e068fc1810, State: Running, Role: LEADER
I20260927 21:09:36.152091 14053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:36.152132 14058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dbb0fc95094c5998f753e068fc1810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37463 } }
I20260927 21:09:36.152490 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53dbb0fc95094c5998f753e068fc1810. Latest consensus state: current_term: 1 leader_uuid: "53dbb0fc95094c5998f753e068fc1810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dbb0fc95094c5998f753e068fc1810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37463 } } }
I20260927 21:09:36.152693 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:36.152626 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53dbb0fc95094c5998f753e068fc1810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dbb0fc95094c5998f753e068fc1810" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37463 } } }
I20260927 21:09:36.153031 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:36.153379 14067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:36.153913 14067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:36.155411 14067 catalog_manager.cc:1347] Generated new cluster ID: 414a483a45064c9bb762f879ea6aa245
I20260927 21:09:36.155460 14067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:09:36.155985 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:36.144966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.0.98.62:37463)
0927 21:09:36.145137 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:36.145148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:36.145181 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:09:36.145708 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:36.145717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:36.145777 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:36.145927 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:36.146482 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:36.147870 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:36.148657 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:36.149273 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:36.149380 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:36.151766 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:36.151772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:36.151773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:36.151785 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:36.153938 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:36.154352 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:36.154355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:36.154355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:36.154440 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:36.155055 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:36.155056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:36.155056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:36.155358 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:36.155502 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:36.155608 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:09:36.155844 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260927 21:09:36.156760 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:09:36.173208 14067 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:09:36.173753 14067 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:36.178839 14067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53dbb0fc95094c5998f753e068fc1810: Generated new TSK 0
I20260927 21:09:36.179010 14067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:09:36.210372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 13985
2026-09-27T21:09:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1314 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:09:36 dist-test-slave-8bgs krb5kdc[14080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:09:36 dist-test-slave-8bgs krb5kdc[14080](info): set up 2 sockets
Sep 27 21:09:36 dist-test-slave-8bgs krb5kdc[14080](info): commencing operation
krb5kdc: starting...
W20260927 21:09:37.231076 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:09:37.342147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38943
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:37.421772 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:37.422014 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:37.422051 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:37.423280 14093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:37.423333 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:37.423357 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:37.423396 14093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:37.423427 14093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:37.424873 14093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:37.425145 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:37.425367 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:37.430029 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:37.430076 14093 server_base.cc:1034] running on GCE node
W20260927 21:09:37.430043 14098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:37.430043 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:37.430540 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:37.430802 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:37.431965 14093 hybrid_clock.cc:648] HybridClock initialized: now 1790543377431951 us; error 26 us; skew 500 ppm
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:37.434414 14093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:37.434792 14093 webserver.cc:460] Webserver started at http://127.0.98.62:42083/ using document root <none> and password file <none>
I20260927 21:09:37.434969 14093 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:37.435009 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:37.435159 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:37.436197 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "13bc302be34c4c1c830e3717f4db5ae4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "14ab5becf72fe183cffc6454549e5381"
server_key_iv: "d59fc647cbb3d79d5ae5061693ddd684"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.436496 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "13bc302be34c4c1c830e3717f4db5ae4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "14ab5becf72fe183cffc6454549e5381"
server_key_iv: "d59fc647cbb3d79d5ae5061693ddd684"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.437700 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:09:37.438362 14108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.438509 14093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:09:37.438592 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "13bc302be34c4c1c830e3717f4db5ae4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "14ab5becf72fe183cffc6454549e5381"
server_key_iv: "d59fc647cbb3d79d5ae5061693ddd684"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.438645 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:37.458755 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:37.461663 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:37.461874 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:37.465651 14093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38943
I20260927 21:09:37.465680 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38943 every 8 connection(s)
I20260927 21:09:37.466032 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:37.466290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14093
I20260927 21:09:37.466416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:09:37.466648 392 external_mini_cluster.cc:1428] Setting key 3e8171c6dd05cba9e5d64e7e7eb479ab
I20260927 21:09:37.467304 14161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:37.469743 14161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap starting.
I20260927 21:09:37.470314 14161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:37.470602 14161 log.cc:824] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:37.471374 14161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: No bootstrap required, opened a new log
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:37.473196 14161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:37.473348 14161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:37.473395 14161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bc302be34c4c1c830e3717f4db5ae4, State: Initialized, Role: FOLLOWER
I20260927 21:09:37.473551 14161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:37.473634 14161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:37.473711 14161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:37.473768 14161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:37.474287 14161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:37.474400 14161 leader_election.cc:302] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13bc302be34c4c1c830e3717f4db5ae4; no voters:
I20260927 21:09:37.474606 14161 leader_election.cc:288] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:09:37.474633 14166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:37.474803 14166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 LEADER]: Becoming Leader. State: Replica: 13bc302be34c4c1c830e3717f4db5ae4, State: Running, Role: LEADER
I20260927 21:09:37.474921 14161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:37.474927 14166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:37.475361 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13bc302be34c4c1c830e3717f4db5ae4. Latest consensus state: current_term: 1 leader_uuid: "13bc302be34c4c1c830e3717f4db5ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } } }
I20260927 21:09:37.475379 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13bc302be34c4c1c830e3717f4db5ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } } }
I20260927 21:09:37.475517 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:37.475528 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:37.475766 14175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:37.476331 14175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:37.477329 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.467181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.0.98.62:38943)
0927 21:09:37.467378 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:37.467386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:37.467434 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:09:37.467903 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:37.467909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:37.467955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:37.468100 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:37.468697 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.469546 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.470375 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.470830 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.470956 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.473217 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:37.473221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:37.473221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:37.473233 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:37.475290 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.475732 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.475734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.475735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.475837 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.476057 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.476058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.476059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.476380 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:37.476926 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:37.476997 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:09:37.477136 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260927 21:09:37.477880 14175 catalog_manager.cc:1347] Generated new cluster ID: 1e961c0586f84bb496b21339f9406261
I20260927 21:09:37.477927 14175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:09:37.499692 14175 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:09:37.500458 14175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:37.504117 14175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Generated new TSK 0
I20260927 21:09:37.504299 14175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:09:37.534269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38943
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:37.613667 14187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:37.613900 14187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:37.613937 14187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:37.615051 14187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:37.615132 14187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:37.615183 14187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:09:37.616588 14187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:37.616853 14187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:37.617069 14187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:37.621851 14192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:37.621945 14187 server_base.cc:1034] running on GCE node
W20260927 21:09:37.621851 14195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:37.621858 14193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:37.622277 14187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:37.622511 14187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:37.623684 14187 hybrid_clock.cc:648] HybridClock initialized: now 1790543377623684 us; error 37 us; skew 500 ppm
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:37.625954 14187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:09:37.626381 14187 webserver.cc:460] Webserver started at http://127.0.98.1:32771/ using document root <none> and password file <none>
I20260927 21:09:37.626643 14187 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:37.626693 14187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:37.626792 14187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:37.627805 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "5d62079001b342e29a3bee36cf17f8e4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "8846f55977e424104807fd8291fc0315"
server_key_iv: "22ef6c7fd750d4b811722f679e6d9517"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.628093 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5d62079001b342e29a3bee36cf17f8e4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "8846f55977e424104807fd8291fc0315"
server_key_iv: "22ef6c7fd750d4b811722f679e6d9517"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.629359 14187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:09:37.630049 14202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.630227 14187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:37.630314 14187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "5d62079001b342e29a3bee36cf17f8e4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "8846f55977e424104807fd8291fc0315"
server_key_iv: "22ef6c7fd750d4b811722f679e6d9517"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.630384 14187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:37.658418 14187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:37.660375 14187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:37.660521 14187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:37.660763 14187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:37.661062 14187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:37.661094 14187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.661118 14187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:37.661160 14187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.667011 14187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43613
I20260927 21:09:37.667109 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43613 every 8 connection(s)
I20260927 21:09:37.667409 14187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:09:37.669236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14187
I20260927 21:09:37.669325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:09:37.669521 392 external_mini_cluster.cc:1428] Setting key a26cdf735dce0e3a622dd7a8bbd6293f
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:37.677680 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.668071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60095 (local address 127.0.98.62:38943)
0927 21:09:37.668116 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:37.668118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:37.668327 (+ 209us) server_negotiation.cc:407] Connection header received
0927 21:09:37.668825 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:37.668826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:37.668838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:37.668888 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:37.669995 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.670716 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.671970 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.672161 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.672215 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.674766 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:37.674768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:37.674768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:37.674774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:37.676239 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.676702 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.676703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.676704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.676788 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.677077 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.677078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.677078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.677175 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:37.677224 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:37.677554 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:09:37.677616 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:09:37.677968 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.668018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:60095)
0927 21:09:37.668174 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:37.668225 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:37.668720 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:37.668965 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:37.668969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:37.669617 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:37.669912 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:37.669919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.670810 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.670811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:37.671889 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:37.671892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.672246 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.672247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:37.672405 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.672944 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:37.672953 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:37.674657 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:37.676316 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:37.676323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:37.676326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:37.676629 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:37.676855 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:37.676856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:37.676856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:37.677009 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:37.677243 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:37.677249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:37.677414 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:09:37.677529 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:09:37.677701 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:09:37.678467 14316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38943
I20260927 21:09:37.678560 14316 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:37.678766 14316 heartbeater.cc:511] Master 127.0.98.62:38943 requested a full tablet report, sending...
I20260927 21:09:37.679252 14125 ts_manager.cc:194] Registered new tserver with Master: 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613)
I20260927 21:09:37.679960 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:09:37.693929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38943
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:37.773533 14321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:37.773739 14321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:37.773763 14321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:37.774890 14321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:37.775012 14321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:37.775076 14321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:09:37.776754 14321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:37.776947 14321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:37.777122 14321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:09:37.781872 14321 server_base.cc:1034] running on GCE node
W20260927 21:09:37.781852 14327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:37.781852 14326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:37.781842 14329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:37.782344 14321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:37.782565 14321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:37.783703 14321 hybrid_clock.cc:648] HybridClock initialized: now 1790543377783691 us; error 34 us; skew 500 ppm
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:37.786226 14321 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:09:37.786669 14321 webserver.cc:460] Webserver started at http://127.0.98.2:43175/ using document root <none> and password file <none>
I20260927 21:09:37.786862 14321 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:37.786903 14321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:37.786988 14321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:37.788043 14321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "af25509c06c3451d9d87f48fdbded8c9"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "08ed2fbe5743d7e95a8308a7c2767525"
server_key_iv: "f69ca5fc488ba9555d6371aef7eada02"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.788342 14321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af25509c06c3451d9d87f48fdbded8c9"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "08ed2fbe5743d7e95a8308a7c2767525"
server_key_iv: "f69ca5fc488ba9555d6371aef7eada02"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.789599 14321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:09:37.790292 14336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.790473 14321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:37.790577 14321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "af25509c06c3451d9d87f48fdbded8c9"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "08ed2fbe5743d7e95a8308a7c2767525"
server_key_iv: "f69ca5fc488ba9555d6371aef7eada02"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.790650 14321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:37.812121 14321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:37.814496 14321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:37.814671 14321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:37.814916 14321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:37.815294 14321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:37.815351 14321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.815394 14321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:37.815428 14321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.821257 14321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44999
I20260927 21:09:37.821301 14449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44999 every 8 connection(s)
I20260927 21:09:37.821619 14321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:37.828527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14321
I20260927 21:09:37.828665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:09:37.828889 392 external_mini_cluster.cc:1428] Setting key 22c705947d69fdc370a9228de85c5f0f
I20260927 21:09:37.830821 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.822290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51521 (local address 127.0.98.62:38943)
0927 21:09:37.822354 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:37.822356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:37.822526 (+ 170us) server_negotiation.cc:407] Connection header received
0927 21:09:37.823017 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:37.823018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:37.823027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:37.823105 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:37.824033 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.824516 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.825439 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.825614 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.825662 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.827816 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:37.827817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:37.827817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:37.827824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:37.829498 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.829883 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.829885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.829885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.829940 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.830188 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.830189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.830189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.830308 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:37.830378 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:37.830656 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 21:09:37.830732 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:09:37.831131 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.822240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:51521)
0927 21:09:37.822390 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:37.822431 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:37.822931 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:37.823222 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:37.823226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:37.823767 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:37.823956 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:37.823961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.824593 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.824593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:37.825366 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:37.825368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.825682 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.825682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:37.825782 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.826199 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:37.826207 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:37.827719 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:37.829560 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:37.829566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:37.829569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:37.829804 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:37.830016 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:37.830017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:37.830017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:37.830120 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:37.830380 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:37.830387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:37.830520 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:09:37.830621 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:09:37.830785 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260927 21:09:37.831571 14450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38943
I20260927 21:09:37.831672 14450 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:37.831844 14450 heartbeater.cc:511] Master 127.0.98.62:38943 requested a full tablet report, sending...
I20260927 21:09:37.832206 14125 ts_manager.cc:194] Registered new tserver with Master: af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999)
I20260927 21:09:37.832584 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51521
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:09:37.853526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38943
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:37.932580 14455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:37.932828 14455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:37.932863 14455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:37.934151 14455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:37.934217 14455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:37.934264 14455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:09:37.935710 14455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:37.935976 14455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:37.936223 14455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:37.941154 14463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:37.941180 14461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:37.941133 14460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:37.941200 14455 server_base.cc:1034] running on GCE node
I20260927 21:09:37.941546 14455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:37.941793 14455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:37.943022 14455 hybrid_clock.cc:648] HybridClock initialized: now 1790543377943013 us; error 43 us; skew 500 ppm
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:37.945577 14455 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:09:37.945991 14455 webserver.cc:460] Webserver started at http://127.0.98.3:44139/ using document root <none> and password file <none>
I20260927 21:09:37.946233 14455 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:37.946302 14455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:37.946455 14455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:37.947660 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "bfeb76ef920579d2c10a051b6a7a2a0f"
server_key_iv: "253e6cdfcca6d994d1bb0811aa514678"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.948016 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "bfeb76ef920579d2c10a051b6a7a2a0f"
server_key_iv: "253e6cdfcca6d994d1bb0811aa514678"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.949329 14455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:09:37.950052 14470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.950220 14455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:37.950320 14455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "bfeb76ef920579d2c10a051b6a7a2a0f"
server_key_iv: "253e6cdfcca6d994d1bb0811aa514678"
server_key_version: "encryptionkey@0"
I20260927 21:09:37.950397 14455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:37.961676 14455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:37.964057 14455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:37.964218 14455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:37.964439 14455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:37.964745 14455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:37.964800 14455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.964843 14455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:37.964871 14455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:37.970434 14455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39971
I20260927 21:09:37.970474 14583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39971 every 8 connection(s)
I20260927 21:09:37.970815 14455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:09:37 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:37.977857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14455
I20260927 21:09:37.977927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:09:37.978119 392 external_mini_cluster.cc:1428] Setting key 95c15cc5b82f53f8eb202f3140500025
I20260927 21:09:37.980180 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.971573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43129 (local address 127.0.98.62:38943)
0927 21:09:37.971631 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:37.971634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:37.971839 (+ 205us) server_negotiation.cc:407] Connection header received
0927 21:09:37.972290 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:37.972291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:37.972301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:37.972356 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:37.973198 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.973746 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.974909 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.975133 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.975183 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.977400 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:37.977402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:37.977402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:37.977408 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:37.979033 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.979421 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.979421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.979422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.979472 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.979688 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.979688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.979689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.979790 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:37.979842 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:37.980065 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:09:37.980120 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:09:37.980315 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.971561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:43129)
0927 21:09:37.971716 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:37.971749 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:37.972200 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:37.972404 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:37.972407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:37.972932 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:37.973122 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:37.973127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.973844 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.973846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:37.974827 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:37.974829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.975201 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.975202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:37.975305 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.975890 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:37.975898 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:37.977315 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:37.979131 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:37.979137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:37.979139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:37.979358 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:37.979516 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:37.979516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:37.979516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:37.979636 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:37.979846 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:37.979851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:37.979965 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 21:09:37.980038 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:09:37.980143 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260927 21:09:37.980739 14584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38943
I20260927 21:09:37.980842 14584 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:37.981001 14584 heartbeater.cc:511] Master 127.0.98.62:38943 requested a full tablet report, sending...
I20260927 21:09:37.981367 14125 ts_manager.cc:194] Registered new tserver with Master: 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971)
I20260927 21:09:37.981746 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43129
I20260927 21:09:37.989825 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:09:37.996963 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:37.991115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.0.98.62:38943)
0927 21:09:37.991161 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:37.991163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:37.991288 (+ 125us) server_negotiation.cc:407] Connection header received
0927 21:09:37.991365 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:37.991366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:37.991376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:37.991431 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:37.991986 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.992459 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.993190 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:37.993428 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:37.993502 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:37.994624 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:37.994626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:37.994635 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:37.994642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:37.995836 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.996170 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.996171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.996171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.996231 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:37.996426 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:37.996427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:37.996427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:37.996571 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:37.996680 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:37.996786 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:09:37.996863 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:09:37.998351 14125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:09:37.999980 14125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:09:38.006484 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.002342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.0.98.3:39971)
0927 21:09:38.002525 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.002530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.002541 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:09:38.002600 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.002602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.002618 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.002711 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.002985 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.004121 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.005349 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.006072 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.006161 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.006223 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.006362 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.006404 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260927 21:09:38.006477 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.002068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:40572)
0927 21:09:38.002244 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.002287 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.002353 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.003067 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.003067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.003069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.003150 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.003151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.004035 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.004036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.005416 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.005417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.006262 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.006263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.006324 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.006327 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.006376 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.006405 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":50,"thread_start_us":54,"threads_started":1}
I20260927 21:09:38.006477 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.002132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39971 (local address 127.0.0.1:43366)
0927 21:09:38.002453 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.002460 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.002492 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.002854 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.002855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.002857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.002898 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.002899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.004225 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.004226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.005265 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.005267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.006163 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.006164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.006242 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.006259 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.006332 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.006389 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":64,"threads_started":1}
I20260927 21:09:38.006839 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.002181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.0.98.1:43613)
0927 21:09:38.002891 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.002896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.002906 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:09:38.002946 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.002948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.002963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.003227 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.003285 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.003933 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.005494 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.006189 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.006253 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.006328 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.006712 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.006764 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":85,"threads_started":1}
I20260927 21:09:38.007282 14518 tablet_service.cc:1467] Processing CreateTablet for tablet 2def85dc04e94d50a3425df54f6695a9 (DEFAULT_TABLE table=test-table [id=d519def79397481fabaa1ad616c242a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:38.007391 14250 tablet_service.cc:1467] Processing CreateTablet for tablet 2def85dc04e94d50a3425df54f6695a9 (DEFAULT_TABLE table=test-table [id=d519def79397481fabaa1ad616c242a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:38.007635 14518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def85dc04e94d50a3425df54f6695a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:38.007644 14250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def85dc04e94d50a3425df54f6695a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:38.007911 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.002132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44999 (local address 127.0.0.1:52648)
0927 21:09:38.002345 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.002354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.002396 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.002742 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.002746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.002748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.002818 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.002819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.004201 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.004202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.005404 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.005407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.007652 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.007653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.007730 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.007734 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.007796 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.007839 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":38,"spinlock_wait_cycles":768,"thread_start_us":73,"threads_started":1}
I20260927 21:09:38.007921 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.002356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.98.2:44999)
0927 21:09:38.002525 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.002530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.002541 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:09:38.002600 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.002602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.002618 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.002719 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.002909 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.004098 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.006365 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.007489 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.007553 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.007613 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.007814 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.007850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
I20260927 21:09:38.008692 14384 tablet_service.cc:1467] Processing CreateTablet for tablet 2def85dc04e94d50a3425df54f6695a9 (DEFAULT_TABLE table=test-table [id=d519def79397481fabaa1ad616c242a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:38.008993 14384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def85dc04e94d50a3425df54f6695a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:38.010869 14598 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap starting.
I20260927 21:09:38.011227 14599 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Bootstrap starting.
I20260927 21:09:38.011622 14598 tablet_bootstrap.cc:654] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:38.011823 14599 tablet_bootstrap.cc:654] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:38.012045 14600 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Bootstrap starting.
I20260927 21:09:38.012212 14598 log.cc:824] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:38.012394 14599 log.cc:824] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:38.012728 14600 tablet_bootstrap.cc:654] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:38.012995 14598 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: No bootstrap required, opened a new log
I20260927 21:09:38.013073 14598 ts_tablet_manager.cc:1397] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:09:38.013180 14600 log.cc:824] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:38.013928 14600 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: No bootstrap required, opened a new log
I20260927 21:09:38.013978 14600 ts_tablet_manager.cc:1397] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:09:38.014593 14598 raft_consensus.cc:348] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.014755 14598 raft_consensus.cc:374] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:38.014801 14598 raft_consensus.cc:729] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d62079001b342e29a3bee36cf17f8e4, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.014915 14598 consensus_queue.cc:260] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.015224 14598 ts_tablet_manager.cc:1428] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:09:38.015246 14316 heartbeater.cc:503] Master 127.0.98.62:38943 was elected leader, sending a full tablet report...
I20260927 21:09:38.015616 14599 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: No bootstrap required, opened a new log
I20260927 21:09:38.015507 14600 raft_consensus.cc:348] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.015662 14600 raft_consensus.cc:374] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:38.015678 14599 ts_tablet_manager.cc:1397] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 21:09:38.015697 14600 raft_consensus.cc:729] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af25509c06c3451d9d87f48fdbded8c9, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.015795 14600 consensus_queue.cc:260] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.016055 14600 ts_tablet_manager.cc:1428] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:09:38.016137 14450 heartbeater.cc:503] Master 127.0.98.62:38943 was elected leader, sending a full tablet report...
I20260927 21:09:38.017426 14599 raft_consensus.cc:348] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.017570 14599 raft_consensus.cc:374] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:38.017616 14599 raft_consensus.cc:729] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b4d640e05d4ddb8c0a2d9f9ea7f894, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.017750 14599 consensus_queue.cc:260] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.018016 14599 ts_tablet_manager.cc:1428] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:09:38.018028 14584 heartbeater.cc:503] Master 127.0.98.62:38943 was elected leader, sending a full tablet report...
W20260927 21:09:38.072196 14451 tablet.cc:2369] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:38.137415 14606 raft_consensus.cc:482] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:38.137598 14606 raft_consensus.cc:504] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.138015 14606 leader_election.cc:288] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613), af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999)
I20260927 21:09:38.141907 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.138325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50047 (local address 127.0.98.1:43613)
0927 21:09:38.138367 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.138369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.138377 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:09:38.138410 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.138411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.138419 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.138468 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.138668 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.139458 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.140986 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.141582 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.141642 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.141650 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.141806 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.141839 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:09:38.141907 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.138287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58419 (local address 127.0.98.2:44999)
0927 21:09:38.138336 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.138338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.138506 (+ 168us) server_negotiation.cc:407] Connection header received
0927 21:09:38.138543 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.138544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.138554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.138596 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.138769 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.139474 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.140985 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.141540 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.141614 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.141627 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.141796 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.141837 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:09:38.141919 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.138207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.98.3:50047)
0927 21:09:38.138258 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.138270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.138318 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.138494 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.138496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.138497 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.138569 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.138575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.139544 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.139545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.140897 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.140899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.141644 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.141645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.141718 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.141721 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.141770 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.141822 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":81}
I20260927 21:09:38.141927 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.138221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44999 (local address 127.0.98.3:58419)
0927 21:09:38.138432 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.138441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.138494 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.138617 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.138617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.138619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.138681 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.138682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.139544 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.139545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.140897 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.140899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.141628 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.141629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.141706 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.141711 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.141770 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.141821 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":149,"thread_start_us":111,"threads_started":1}
I20260927 21:09:38.142342 14270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def85dc04e94d50a3425df54f6695a9" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d62079001b342e29a3bee36cf17f8e4" is_pre_election: true
I20260927 21:09:38.142455 14404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def85dc04e94d50a3425df54f6695a9" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25509c06c3451d9d87f48fdbded8c9" is_pre_election: true
I20260927 21:09:38.142557 14270 raft_consensus.cc:2454] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 0.
I20260927 21:09:38.142630 14404 raft_consensus.cc:2454] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 0.
I20260927 21:09:38.142779 14471 leader_election.cc:302] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b4d640e05d4ddb8c0a2d9f9ea7f894, 5d62079001b342e29a3bee36cf17f8e4; no voters:
I20260927 21:09:38.142921 14606 raft_consensus.cc:2781] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:09:38.142985 14606 raft_consensus.cc:482] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:38.143018 14606 raft_consensus.cc:3037] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:38.143740 14606 raft_consensus.cc:504] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.143882 14606 leader_election.cc:288] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 election: Requested vote from peers 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613), af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999)
I20260927 21:09:38.144027 14404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def85dc04e94d50a3425df54f6695a9" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25509c06c3451d9d87f48fdbded8c9"
I20260927 21:09:38.144104 14404 raft_consensus.cc:3037] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:38.144785 14404 raft_consensus.cc:2454] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 1.
I20260927 21:09:38.144899 14270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def85dc04e94d50a3425df54f6695a9" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d62079001b342e29a3bee36cf17f8e4"
I20260927 21:09:38.144940 14473 leader_election.cc:302] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b4d640e05d4ddb8c0a2d9f9ea7f894, af25509c06c3451d9d87f48fdbded8c9; no voters:
I20260927 21:09:38.144979 14270 raft_consensus.cc:3037] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:38.145015 14606 raft_consensus.cc:2781] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:38.145143 14606 raft_consensus.cc:686] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 LEADER]: Becoming Leader. State: Replica: 11b4d640e05d4ddb8c0a2d9f9ea7f894, State: Running, Role: LEADER
I20260927 21:09:38.145238 14606 consensus_queue.cc:237] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.145864 14270 raft_consensus.cc:2454] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 1.
I20260927 21:09:38.146096 14125 catalog_manager.cc:5495] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } health_report { overall_health: UNKNOWN } } }
W20260927 21:09:38.168143 14317 tablet.cc:2369] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:38.185389 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.183292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.0.98.3:39971)
0927 21:09:38.183348 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.183350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.183357 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:09:38.183391 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.183392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.183409 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.183469 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:09:38.183583 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.184044 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.184731 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.184900 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.184946 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.185060 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:09:38.185172 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:09:38.185234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.185272 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.185287 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:09:38.187330 14404 raft_consensus.cc:1261] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Refusing update from remote peer 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:38.187330 14270 raft_consensus.cc:1261] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Refusing update from remote peer 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:38.187713 14608 consensus_queue.cc:1035] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:38.187865 14606 consensus_queue.cc:1035] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:38.190755 14611 mvcc.cc:205] Tried to move back new op lower bound from 7334065677052268544 to 7334065676883550208. Current Snapshot: MvccSnapshot[applied={T|T < 7334065677052268544 or (T in {7334065677052268544})}]
I20260927 21:09:38.190877 14613 mvcc.cc:205] Tried to move back new op lower bound from 7334065677052268544 to 7334065676883550208. Current Snapshot: MvccSnapshot[applied={T|T < 7334065677052268544 or (T in {7334065677052268544})}]
I20260927 21:09:38.192543 14124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:38.192643 14124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:38.193395 14124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Sending DeleteTablet for 3 replicas of tablet 2def85dc04e94d50a3425df54f6695a9
I20260927 21:09:38.193760 14250 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def85dc04e94d50a3425df54f6695a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40572
I20260927 21:09:38.193801 14384 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def85dc04e94d50a3425df54f6695a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52648
I20260927 21:09:38.193976 14621 tablet_replica.cc:330] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: stopping tablet replica
I20260927 21:09:38.193980 14518 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def85dc04e94d50a3425df54f6695a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43366
I20260927 21:09:38.194075 14621 raft_consensus.cc:2229] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:38.194185 14621 raft_consensus.cc:2258] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:38.194217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14187
I20260927 21:09:38.194199 14623 tablet_replica.cc:330] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: stopping tablet replica
I20260927 21:09:38.194984 14623 raft_consensus.cc:2229] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:09:38.195178 14623 raft_consensus.cc:2258] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:38.195398 14622 tablet_replica.cc:330] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: stopping tablet replica
I20260927 21:09:38.195503 14622 raft_consensus.cc:2229] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:38.195608 14622 raft_consensus.cc:2258] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:38.196075 14622 ts_tablet_manager.cc:1905] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:38.197214 14623 ts_tablet_manager.cc:1905] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:38.198009 14622 ts_tablet_manager.cc:1918] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:38.198086 14622 log.cc:1196] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/2def85dc04e94d50a3425df54f6695a9
I20260927 21:09:38.198362 14622 ts_tablet_manager.cc:1939] T 2def85dc04e94d50a3425df54f6695a9 P af25509c06c3451d9d87f48fdbded8c9: Deleting consensus metadata
I20260927 21:09:38.198777 14111 catalog_manager.cc:4841] TS af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999): tablet 2def85dc04e94d50a3425df54f6695a9 (table test-table [id=d519def79397481fabaa1ad616c242a3]) successfully deleted
I20260927 21:09:38.198839 14623 ts_tablet_manager.cc:1918] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:38.198892 14623 log.cc:1196] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/2def85dc04e94d50a3425df54f6695a9
I20260927 21:09:38.199183 14623 ts_tablet_manager.cc:1939] T 2def85dc04e94d50a3425df54f6695a9 P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Deleting consensus metadata
I20260927 21:09:38.199574 14112 catalog_manager.cc:4841] TS 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971): tablet 2def85dc04e94d50a3425df54f6695a9 (table test-table [id=d519def79397481fabaa1ad616c242a3]) successfully deleted
W20260927 21:09:38.202450 14109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43613 (error 108)
I20260927 21:09:38.202983 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.202839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:40580)
0927 21:09:38.202902 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.202944 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 21:09:38.203017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14321
W20260927 21:09:38.207195 14109 catalog_manager.cc:4584] TS 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2def85dc04e94d50a3425df54f6695a9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43613: connect: Connection refused (error 111)
I20260927 21:09:38.210459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14455
I20260927 21:09:38.214788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14093
I20260927 21:09:38.219870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38943
--webserver_interface=127.0.98.62
--webserver_port=42083
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:38.301784 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:38.302031 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:38.302067 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:38.303325 14625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:38.303385 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:38.303408 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:38.303437 14625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:38.303488 14625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:38.304963 14625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:38.305235 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:38.305459 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:38.313838 14631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.313970 14625 server_base.cc:1034] running on GCE node
W20260927 21:09:38.313812 14634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:38.313830 14632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.314330 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:38.314640 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:38.315799 14625 hybrid_clock.cc:648] HybridClock initialized: now 1790543378315788 us; error 29 us; skew 500 ppm
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:38.322064 14625 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:38.322510 14625 webserver.cc:460] Webserver started at http://127.0.98.62:42083/ using document root <none> and password file <none>
I20260927 21:09:38.322773 14625 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:38.322854 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:38.324318 14625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:38.324950 14641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.325110 14625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:38.325214 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "13bc302be34c4c1c830e3717f4db5ae4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "14ab5becf72fe183cffc6454549e5381"
server_key_iv: "d59fc647cbb3d79d5ae5061693ddd684"
server_key_version: "encryptionkey@0"
I20260927 21:09:38.325610 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:38.340363 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:38.342639 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:38.342803 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:38.346740 14625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38943
I20260927 21:09:38.346776 14693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38943 every 8 connection(s)
I20260927 21:09:38.347196 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:38.348781 14694 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:09:38.349361 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap starting.
I20260927 21:09:38.354420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14625
I20260927 21:09:38.354693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38943
--webserver_port=32771
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:38.357826 14694 log.cc:824] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:38.359817 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:38.360097 14694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap complete.
I20260927 21:09:38.362380 14694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:38.362613 14694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bc302be34c4c1c830e3717f4db5ae4, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.362789 14694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:38.362849 14694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:38.362911 14694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:38.362957 14694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:09:38.364018 14694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:38.364106 14694 leader_election.cc:302] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13bc302be34c4c1c830e3717f4db5ae4; no voters:
I20260927 21:09:38.364310 14694 leader_election.cc:288] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:09:38.364344 14698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:09:38.364454 14698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 LEADER]: Becoming Leader. State: Replica: 13bc302be34c4c1c830e3717f4db5ae4, State: Running, Role: LEADER
I20260927 21:09:38.364586 14694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:38.364600 14698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:38.365084 14699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "13bc302be34c4c1c830e3717f4db5ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } } }
I20260927 21:09:38.365168 14699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:38.365501 14704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:38.365556 14700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13bc302be34c4c1c830e3717f4db5ae4. Latest consensus state: current_term: 2 leader_uuid: "13bc302be34c4c1c830e3717f4db5ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } } }
I20260927 21:09:38.365688 14700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:38.366295 14704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:38.366390 14704 catalog_manager.cc:1259] Loaded cluster ID: 1e961c0586f84bb496b21339f9406261
I20260927 21:09:38.366464 14704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:09:38.366856 14704 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 21:09:38.371938 14704 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 21:09:38.374922 14704 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:38.375129 14704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Loaded TSK: 0
I20260927 21:09:38.375347 14704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:09:38.438376 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:38.438581 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:38.438604 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:38.439779 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:38.439819 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:38.439845 14696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:09:38.441256 14696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32771
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:38.441483 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:38.441721 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:38.446477 14721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:38.446450 14722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:38.446779 14724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.446787 14696 server_base.cc:1034] running on GCE node
I20260927 21:09:38.447008 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:38.447283 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:38.448437 14696 hybrid_clock.cc:648] HybridClock initialized: now 1790543378448411 us; error 37 us; skew 500 ppm
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:38.451066 14696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:09:38.451516 14696 webserver.cc:460] Webserver started at http://127.0.98.1:32771/ using document root <none> and password file <none>
I20260927 21:09:38.451733 14696 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:38.451813 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:38.453147 14696 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:38.453847 14731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.454027 14696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:38.454123 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "5d62079001b342e29a3bee36cf17f8e4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "8846f55977e424104807fd8291fc0315"
server_key_iv: "22ef6c7fd750d4b811722f679e6d9517"
server_key_version: "encryptionkey@0"
I20260927 21:09:38.454471 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:38.483577 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:38.485812 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:38.485981 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:38.486224 14696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:38.486776 14738 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:09:38.488668 14696 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:09:38.488745 14696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:09:38.488791 14696 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:09:38.489441 14696 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:09:38.489496 14696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:09:38.489575 14738 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap starting.
I20260927 21:09:38.496347 14696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43613
I20260927 21:09:38.496424 14845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43613 every 8 connection(s)
I20260927 21:09:38.496742 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:09:38.500068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14696
I20260927 21:09:38.500386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44999
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38943
--webserver_port=43175
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:38.500937 14738 log.cc:824] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:38.506537 14738 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:38.506897 14738 tablet_bootstrap.cc:492] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap complete.
I20260927 21:09:38.508204 14738 ts_tablet_manager.cc:1397] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20260927 21:09:38.509446 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.497586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41667 (local address 127.0.98.62:38943)
0927 21:09:38.497752 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.497761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.497847 (+ 86us) server_negotiation.cc:407] Connection header received
0927 21:09:38.498699 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.498705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.498755 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.498865 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:38.499924 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.501348 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.502922 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.503360 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.503493 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.505943 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:38.505946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:38.505946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:38.505957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:38.507713 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.508133 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.508135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.508136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.508228 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.508523 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.508524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.508524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.508748 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:38.508839 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.509132 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.509255 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260927 21:09:38.509446 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.497491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:41667)
0927 21:09:38.497694 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.497739 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.498473 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.498949 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.498954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.499456 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:38.499769 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.499776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.501465 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.501466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.502834 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.502837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.503474 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.503476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.503612 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.504203 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:38.504213 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:38.505818 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:38.507808 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:38.507814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:38.507816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:38.508032 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:38.508348 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:38.508349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:38.508349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:38.508450 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:38.508829 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:38.508835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:38.508971 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.509078 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.509255 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
I20260927 21:09:38.509919 14738 raft_consensus.cc:348] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.510325 14738 raft_consensus.cc:729] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d62079001b342e29a3bee36cf17f8e4, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.510528 14846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38943
I20260927 21:09:38.510511 14738 consensus_queue.cc:260] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } }
I20260927 21:09:38.510622 14846 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:38.510792 14846 heartbeater.cc:511] Master 127.0.98.62:38943 requested a full tablet report, sending...
I20260927 21:09:38.510821 14738 ts_tablet_manager.cc:1428] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260927 21:09:38.511502 14658 ts_manager.cc:194] Registered new tserver with Master: 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613)
I20260927 21:09:38.512482 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41667
I20260927 21:09:38.513257 14846 heartbeater.cc:503] Master 127.0.98.62:38943 was elected leader, sending a full tablet report...
I20260927 21:09:38.518512 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.512227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.0.98.1:43613)
0927 21:09:38.515380 (+ 3153us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.515383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.515390 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:09:38.515414 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.515415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.515425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.515480 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.515658 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.516278 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.517745 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.518170 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.518227 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.518259 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.518438 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.518465 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3128,"thread_start_us":74,"threads_started":1}
I20260927 21:09:38.518550 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.512164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:40592)
0927 21:09:38.515200 (+ 3036us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.515229 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.515283 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.515512 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.515516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.515519 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.515585 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.515586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.516351 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.516352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.517658 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.517662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.518250 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.518251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.518335 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.518348 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.518422 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.518475 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3011,"thread_start_us":81,"threads_started":1}
I20260927 21:09:38.519016 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def85dc04e94d50a3425df54f6695a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40592
I20260927 21:09:38.519526 14855 tablet_replica.cc:330] stopping tablet replica
I20260927 21:09:38.519636 14855 raft_consensus.cc:2229] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:38.519701 14855 raft_consensus.cc:2258] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:38.519972 14855 ts_tablet_manager.cc:1905] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:38.521986 14855 ts_tablet_manager.cc:1918] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:38.522091 14855 log.cc:1196] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/2def85dc04e94d50a3425df54f6695a9
I20260927 21:09:38.522313 14855 ts_tablet_manager.cc:1939] T 2def85dc04e94d50a3425df54f6695a9 P 5d62079001b342e29a3bee36cf17f8e4: Deleting consensus metadata
I20260927 21:09:38.522845 14642 catalog_manager.cc:4841] TS 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613): tablet 2def85dc04e94d50a3425df54f6695a9 (table test-table [id=d519def79397481fabaa1ad616c242a3]) successfully deleted
W20260927 21:09:38.584760 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:38.584951 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:38.584975 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:38.586024 14851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:38.586058 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:38.586083 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:09:38.587474 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43175
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:38.587651 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:38.587816 14851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:38.592682 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:38.592660 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.592728 14851 server_base.cc:1034] running on GCE node
W20260927 21:09:38.592656 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.593171 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:38.593391 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:38.594552 14851 hybrid_clock.cc:648] HybridClock initialized: now 1790543378594526 us; error 42 us; skew 500 ppm
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:38.597015 14851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:09:38.597436 14851 webserver.cc:460] Webserver started at http://127.0.98.2:43175/ using document root <none> and password file <none>
I20260927 21:09:38.597607 14851 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:38.597656 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:38.598865 14851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:38.599424 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.599612 14851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.599696 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "af25509c06c3451d9d87f48fdbded8c9"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "08ed2fbe5743d7e95a8308a7c2767525"
server_key_iv: "f69ca5fc488ba9555d6371aef7eada02"
server_key_version: "encryptionkey@0"
I20260927 21:09:38.600014 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:38.612613 14851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:38.614815 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:38.614920 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:38.615183 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:38.615482 14851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:38.615516 14851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.615558 14851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:38.615588 14851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.620981 14851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44999
I20260927 21:09:38.621055 14983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44999 every 8 connection(s)
I20260927 21:09:38.621381 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:09:38.625911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14851
I20260927 21:09:38.626178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38943
--webserver_port=44139
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:38.630746 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.622073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:46681)
0927 21:09:38.622249 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.622283 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.622750 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.622946 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.622949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.623557 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:38.623744 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.623749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.624378 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.624378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.625147 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.625149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.625480 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.625480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.625572 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.626127 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:38.626134 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:38.627590 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:38.629449 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:38.629455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:38.629456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:38.629660 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:38.629900 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:38.629900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:38.629901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:38.630001 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:38.630290 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:38.630294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:38.630399 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.630475 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.630585 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260927 21:09:38.630767 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.622101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46681 (local address 127.0.98.62:38943)
0927 21:09:38.622181 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.622184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.622368 (+ 184us) server_negotiation.cc:407] Connection header received
0927 21:09:38.622834 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.622835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.622844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.622895 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:38.623821 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.624303 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.625219 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.625410 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.625459 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.627739 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:38.627740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:38.627741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:38.627750 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:38.629377 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.629744 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.629746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.629746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.629817 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.630069 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.630070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.630071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.630221 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:38.630299 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.630522 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.630615 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:09:38.631331 14984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38943
I20260927 21:09:38.631448 14984 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:38.631634 14984 heartbeater.cc:511] Master 127.0.98.62:38943 requested a full tablet report, sending...
I20260927 21:09:38.632067 14658 ts_manager.cc:194] Registered new tserver with Master: af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999)
I20260927 21:09:38.632608 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46681
W20260927 21:09:38.706645 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:38.706840 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:38.706863 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:38.707979 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:38.708020 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:38.708045 14987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:09:38.709425 14987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44139
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:38.709611 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:38.709780 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:38.714529 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:38.714618 14993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.714636 14987 server_base.cc:1034] running on GCE node
W20260927 21:09:38.714538 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:38.714866 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:38.715054 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:38.716233 14987 hybrid_clock.cc:648] HybridClock initialized: now 1790543378716214 us; error 34 us; skew 500 ppm
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:38.718534 14987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:09:38.718971 14987 webserver.cc:460] Webserver started at http://127.0.98.3:44139/ using document root <none> and password file <none>
I20260927 21:09:38.719225 14987 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:38.719286 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:38.720588 14987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:38.721129 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.721333 14987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:38.721417 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "bfeb76ef920579d2c10a051b6a7a2a0f"
server_key_iv: "253e6cdfcca6d994d1bb0811aa514678"
server_key_version: "encryptionkey@0"
I20260927 21:09:38.721737 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:38.736720 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:38.738507 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:38.738672 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:38.738897 14987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:38.739241 14987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:38.739281 14987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.739303 14987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:38.739347 14987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.745088 14987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39971
I20260927 21:09:38.745134 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39971 every 8 connection(s)
I20260927 21:09:38.745455 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:09:38 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:38.751530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 14987
I20260927 21:09:38.754812 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.746139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42039 (local address 127.0.98.62:38943)
0927 21:09:38.746188 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.746191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.746372 (+ 181us) server_negotiation.cc:407] Connection header received
0927 21:09:38.746886 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.746887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.746900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.746946 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:38.747988 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.748476 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.749465 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.749664 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.749757 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.751844 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:38.751845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:38.751846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:38.751853 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:38.753451 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.753885 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.753886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.753886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.753934 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.754190 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.754190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.754191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.754293 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:38.754342 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.754646 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.754718 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:09:38.754940 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.746107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:42039)
0927 21:09:38.746252 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.746291 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.746783 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.747041 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.747044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.747670 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:38.747855 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.747859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.748583 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.748585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.749373 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.749375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.749765 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.749766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.749894 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.750393 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:38.750401 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:38.751753 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:38.753531 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:38.753536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:38.753538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:38.753814 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:38.753997 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:38.753997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:38.753998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:38.754128 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:38.754353 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:38.754358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:38.754505 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.754612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.754746 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260927 21:09:38.755401 15116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38943
I20260927 21:09:38.755509 15116 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:38.755725 15116 heartbeater.cc:511] Master 127.0.98.62:38943 requested a full tablet report, sending...
I20260927 21:09:38.756119 14658 ts_manager.cc:194] Registered new tserver with Master: 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971)
I20260927 21:09:38.756610 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42039
I20260927 21:09:38.758787 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.751936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.0.98.62:38943)
0927 21:09:38.752080 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.752082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.752091 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:09:38.752112 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.752113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.752124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.752183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:38.752867 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.753623 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.754628 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.754936 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.755001 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.756415 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:38.756416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:38.756417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:38.756423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:38.757918 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.758208 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.758209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.758209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.758255 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:38.758394 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:38.758394 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:38.758395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:38.758473 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:38.758573 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.758621 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.758676 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260927 21:09:38.759421 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:09:38.761754 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.759711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.0.98.62:38943)
0927 21:09:38.759763 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.759764 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.759870 (+ 106us) server_negotiation.cc:407] Connection header received
0927 21:09:38.759910 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.759911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.759935 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.759985 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:09:38.760055 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.760484 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.761165 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.761346 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.761393 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.761515 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:09:38.761597 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:09:38.761653 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.761694 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.761712 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:09:38.761976 14658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:09:38.763579 14658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:09:38.765594 14780 tablet_service.cc:1467] Processing CreateTablet for tablet 66c566ee59e44ff7a506a8058dc0c73d (DEFAULT_TABLE table=test-table [id=35a3ca9e8d10493985b7ff83bae7fb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:38.765841 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c566ee59e44ff7a506a8058dc0c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:38.767323 14738 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap starting.
I20260927 21:09:38.767663 14738 tablet_bootstrap.cc:654] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:38.768189 14738 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: No bootstrap required, opened a new log
I20260927 21:09:38.768240 14738 ts_tablet_manager.cc:1397] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:09:38.768347 14738 raft_consensus.cc:348] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.768415 14738 raft_consensus.cc:374] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:38.768452 14738 raft_consensus.cc:729] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d62079001b342e29a3bee36cf17f8e4, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.768518 14738 consensus_queue.cc:260] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.768625 14738 ts_tablet_manager.cc:1428] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:38.769153 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.765311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44999 (local address 127.0.0.1:52654)
0927 21:09:38.765355 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.765365 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.765398 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.765784 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.765785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.765787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.765824 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.765825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.767079 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.767080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.768047 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.768049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.768946 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.768947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.769010 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.769014 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.769045 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.769088 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:09:38.769470 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.765404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.98.2:44999)
0927 21:09:38.765552 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.765556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.765567 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:09:38.765606 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.765608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.765624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.765899 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.765972 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.766976 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.768136 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.768854 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.769265 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.769330 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.769355 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.769393 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260927 21:09:38.770135 14918 tablet_service.cc:1467] Processing CreateTablet for tablet 66c566ee59e44ff7a506a8058dc0c73d (DEFAULT_TABLE table=test-table [id=35a3ca9e8d10493985b7ff83bae7fb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:38.770490 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c566ee59e44ff7a506a8058dc0c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:38.771199 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.765333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39971 (local address 127.0.0.1:43390)
0927 21:09:38.767412 (+ 2079us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.767422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.767484 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.767687 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.767688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.767690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.767729 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.767731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.769065 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.769065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.770067 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.770070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.770928 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.770929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.770986 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.770990 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.771052 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.771133 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"thread_start_us":79,"threads_started":1}
I20260927 21:09:38.771219 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.765551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.98.3:39971)
0927 21:09:38.765692 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.765696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.767541 (+ 1845us) server_negotiation.cc:407] Connection header received
0927 21:09:38.767581 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.767583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.767598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.767677 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.767813 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.768970 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.770146 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.770849 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.770906 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.770937 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.771067 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.771144 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260927 21:09:38.771782 15050 tablet_service.cc:1467] Processing CreateTablet for tablet 66c566ee59e44ff7a506a8058dc0c73d (DEFAULT_TABLE table=test-table [id=35a3ca9e8d10493985b7ff83bae7fb47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:38.772054 15050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c566ee59e44ff7a506a8058dc0c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:38.773412 15126 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Bootstrap starting.
I20260927 21:09:38.774061 15126 tablet_bootstrap.cc:654] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:38.774575 15126 log.cc:824] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:38.774916 15127 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Bootstrap starting.
I20260927 21:09:38.775334 15126 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: No bootstrap required, opened a new log
I20260927 21:09:38.775417 15126 ts_tablet_manager.cc:1397] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:09:38.775589 15127 tablet_bootstrap.cc:654] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:38.776189 15127 log.cc:824] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:38.777128 15127 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: No bootstrap required, opened a new log
I20260927 21:09:38.777215 15127 ts_tablet_manager.cc:1397] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:09:38.777426 15126 raft_consensus.cc:348] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.777558 15126 raft_consensus.cc:374] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:38.777580 15126 raft_consensus.cc:729] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af25509c06c3451d9d87f48fdbded8c9, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.777693 15126 consensus_queue.cc:260] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.777962 15126 ts_tablet_manager.cc:1428] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:09:38.777987 14984 heartbeater.cc:503] Master 127.0.98.62:38943 was elected leader, sending a full tablet report...
I20260927 21:09:38.779161 15127 raft_consensus.cc:348] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.779313 15127 raft_consensus.cc:374] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:38.779357 15127 raft_consensus.cc:729] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b4d640e05d4ddb8c0a2d9f9ea7f894, State: Initialized, Role: FOLLOWER
I20260927 21:09:38.779460 15127 consensus_queue.cc:260] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.779749 15127 ts_tablet_manager.cc:1428] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:09:38.779763 15116 heartbeater.cc:503] Master 127.0.98.62:38943 was elected leader, sending a full tablet report...
W20260927 21:09:38.871928 14985 tablet.cc:2369] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:38.935207 15131 raft_consensus.cc:482] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:38.935395 15131 raft_consensus.cc:504] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.935808 15131 leader_election.cc:288] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999), 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613)
I20260927 21:09:38.939679 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.936082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54589 (local address 127.0.98.2:44999)
0927 21:09:38.936132 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.936135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.936145 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:09:38.936177 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.936178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.936189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.936261 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.936446 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.937226 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.938734 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.939320 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.939388 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.939398 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.939567 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.939605 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:09:38.939679 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.936129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50811 (local address 127.0.98.1:43613)
0927 21:09:38.936171 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.936173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.936318 (+ 145us) server_negotiation.cc:407] Connection header received
0927 21:09:38.936372 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.936373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.936383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.936458 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:38.936600 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.937281 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.938733 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.939310 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.939388 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.939398 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.939564 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.939606 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:09:38.939743 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.935991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44999 (local address 127.0.98.3:54589)
0927 21:09:38.936033 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.936045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.936094 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.936280 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.936281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.936283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.936357 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.936359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.937341 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.937342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.938623 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.938626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.939399 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.939400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.939483 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.939485 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.939545 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.939606 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":52}
I20260927 21:09:38.939749 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.936054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.98.3:50811)
0927 21:09:38.936257 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:38.936267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:38.936318 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:38.936471 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:38.936472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:38.936473 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:38.936527 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.936527 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.937350 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.937351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.938623 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:38.938626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.939399 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.939400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:38.939480 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.939485 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:09:38.939545 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:09:38.939606 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":56,"thread_start_us":119,"threads_started":1}
I20260927 21:09:38.940112 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c566ee59e44ff7a506a8058dc0c73d" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25509c06c3451d9d87f48fdbded8c9" is_pre_election: true
I20260927 21:09:38.940186 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c566ee59e44ff7a506a8058dc0c73d" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d62079001b342e29a3bee36cf17f8e4" is_pre_election: true
I20260927 21:09:38.940280 14938 raft_consensus.cc:2454] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 0.
I20260927 21:09:38.940395 14800 raft_consensus.cc:2454] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 0.
I20260927 21:09:38.940533 15005 leader_election.cc:302] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b4d640e05d4ddb8c0a2d9f9ea7f894, af25509c06c3451d9d87f48fdbded8c9; no voters:
I20260927 21:09:38.940671 15131 raft_consensus.cc:2781] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:09:38.940725 15131 raft_consensus.cc:482] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:38.940761 15131 raft_consensus.cc:3037] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:38.941442 15131 raft_consensus.cc:504] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.941577 15131 leader_election.cc:288] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 election: Requested vote from peers af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999), 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613)
I20260927 21:09:38.941744 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c566ee59e44ff7a506a8058dc0c73d" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25509c06c3451d9d87f48fdbded8c9"
I20260927 21:09:38.941742 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c566ee59e44ff7a506a8058dc0c73d" candidate_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d62079001b342e29a3bee36cf17f8e4"
I20260927 21:09:38.941828 14800 raft_consensus.cc:3037] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:38.941828 14938 raft_consensus.cc:3037] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:38.942787 14938 raft_consensus.cc:2454] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 1.
I20260927 21:09:38.942955 15005 leader_election.cc:302] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b4d640e05d4ddb8c0a2d9f9ea7f894, af25509c06c3451d9d87f48fdbded8c9; no voters:
I20260927 21:09:38.943061 15131 raft_consensus.cc:2781] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:38.943200 14800 raft_consensus.cc:2454] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b4d640e05d4ddb8c0a2d9f9ea7f894 in term 1.
I20260927 21:09:38.943269 15131 raft_consensus.cc:686] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 LEADER]: Becoming Leader. State: Replica: 11b4d640e05d4ddb8c0a2d9f9ea7f894, State: Running, Role: LEADER
I20260927 21:09:38.943374 15131 consensus_queue.cc:237] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:38.944242 14658 catalog_manager.cc:5495] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } health_report { overall_health: UNKNOWN } } }
I20260927 21:09:38.994326 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:38.991947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.0.98.3:39971)
0927 21:09:38.992014 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:38.992016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:38.992025 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:09:38.992069 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:38.992070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:38.992094 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:38.992186 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:09:38.992303 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.992783 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.993516 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:38.993723 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:38.993799 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:38.993962 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:09:38.994076 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:09:38.994167 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:38.994189 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 21:09:38.994210 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 21:09:38.995893 15117 tablet.cc:2369] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:38.996271 14800 raft_consensus.cc:1261] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Refusing update from remote peer 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:38.996469 14938 raft_consensus.cc:1261] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Refusing update from remote peer 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:38.996611 15131 consensus_queue.cc:1035] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:38.996778 15133 consensus_queue.cc:1035] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:09:38.997341 14847 tablet.cc:2369] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:38.998570 15135 mvcc.cc:205] Tried to move back new op lower bound from 7334065680365592576 to 7334065680152752128. Current Snapshot: MvccSnapshot[applied={T|T < 7334065680365592576}]
I20260927 21:09:38.999017 15136 mvcc.cc:205] Tried to move back new op lower bound from 7334065680365592576 to 7334065680152752128. Current Snapshot: MvccSnapshot[applied={T|T < 7334065680365592576}]
I20260927 21:09:39.000720 14658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:39.000808 14658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:39.001432 14658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Sending DeleteTablet for 3 replicas of tablet 66c566ee59e44ff7a506a8058dc0c73d
I20260927 21:09:39.001761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14696
I20260927 21:09:39.001758 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c566ee59e44ff7a506a8058dc0c73d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40592
I20260927 21:09:39.001760 15050 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c566ee59e44ff7a506a8058dc0c73d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43390
I20260927 21:09:39.001829 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c566ee59e44ff7a506a8058dc0c73d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52654
W20260927 21:09:39.006662 14642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43613 (error 108)
I20260927 21:09:39.006722 15145 tablet_replica.cc:330] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: stopping tablet replica
I20260927 21:09:39.006810 15145 raft_consensus.cc:2229] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:09:39.006915 15145 raft_consensus.cc:2258] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:39.007074 15146 tablet_replica.cc:330] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: stopping tablet replica
I20260927 21:09:39.007192 15146 raft_consensus.cc:2229] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:39.007294 15146 raft_consensus.cc:2258] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:39.007346 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.007232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.0.1:40598)
0927 21:09:39.007279 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.007313 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:09:39.007570 15145 ts_tablet_manager.cc:1905] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:39.007736 15146 ts_tablet_manager.cc:1905] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:09:39.007453 14642 catalog_manager.cc:4584] TS 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66c566ee59e44ff7a506a8058dc0c73d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43613: connect: Connection refused (error 111)
I20260927 21:09:39.008037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14851
I20260927 21:09:39.010202 15145 ts_tablet_manager.cc:1918] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:39.010277 15145 log.cc:1196] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/66c566ee59e44ff7a506a8058dc0c73d
I20260927 21:09:39.010541 15145 ts_tablet_manager.cc:1939] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Deleting consensus metadata
I20260927 21:09:39.010972 14645 catalog_manager.cc:4841] TS 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971): tablet 66c566ee59e44ff7a506a8058dc0c73d (table test-table [id=35a3ca9e8d10493985b7ff83bae7fb47]) successfully deleted
I20260927 21:09:39.012785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14987
I20260927 21:09:39.012807 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.012715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44999 (local address 127.0.0.1:52662)
0927 21:09:39.012753 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.012783 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 21:09:39.017999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 14625
I20260927 21:09:39.023064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38943
--webserver_interface=127.0.98.62
--webserver_port=42083
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:39.102483 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:39.102698 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:39.102720 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:39.103909 15148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:39.103946 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:39.103961 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:39.103997 15148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:39.104023 15148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:39.105389 15148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:39.105611 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:39.105845 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:39.110337 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:39.110324 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.110339 15148 server_base.cc:1034] running on GCE node
W20260927 21:09:39.110325 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.110806 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:39.111081 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:39.112267 15148 hybrid_clock.cc:648] HybridClock initialized: now 1790543379112261 us; error 30 us; skew 500 ppm
Sep 27 21:09:39 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:39.114543 15148 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:39.114895 15148 webserver.cc:460] Webserver started at http://127.0.98.62:42083/ using document root <none> and password file <none>
I20260927 21:09:39.115059 15148 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:39.115150 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:39.116395 15148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:09:39.117019 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:39.117213 15148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:09:39.117301 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "13bc302be34c4c1c830e3717f4db5ae4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "14ab5becf72fe183cffc6454549e5381"
server_key_iv: "d59fc647cbb3d79d5ae5061693ddd684"
server_key_version: "encryptionkey@0"
I20260927 21:09:39.117606 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:39.131417 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:39.133272 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:39.133445 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:39.139710 15148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38943
I20260927 21:09:39.139804 15215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38943 every 8 connection(s)
I20260927 21:09:39.140185 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:39.142269 15216 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:09:39.143160 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap starting.
I20260927 21:09:39.147778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15148
I20260927 21:09:39.148096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38943
--webserver_port=32771
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:39.153277 15216 log.cc:824] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:39.155797 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:39.156097 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: Bootstrap complete.
I20260927 21:09:39.159912 15216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:39.160297 15216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bc302be34c4c1c830e3717f4db5ae4, State: Initialized, Role: FOLLOWER
I20260927 21:09:39.160471 15216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:39.160554 15216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:39.160604 15216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:39.160655 15216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:09:39.161775 15216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:39.161900 15216 leader_election.cc:302] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13bc302be34c4c1c830e3717f4db5ae4; no voters:
I20260927 21:09:39.162062 15216 leader_election.cc:288] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:09:39.162098 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:09:39.162209 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [term 3 LEADER]: Becoming Leader. State: Replica: 13bc302be34c4c1c830e3717f4db5ae4, State: Running, Role: LEADER
I20260927 21:09:39.162286 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } }
I20260927 21:09:39.162338 15216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:39.162531 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13bc302be34c4c1c830e3717f4db5ae4. Latest consensus state: current_term: 3 leader_uuid: "13bc302be34c4c1c830e3717f4db5ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } } }
I20260927 21:09:39.162611 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:39.162575 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "13bc302be34c4c1c830e3717f4db5ae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bc302be34c4c1c830e3717f4db5ae4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38943 } } }
I20260927 21:09:39.162779 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:39.164168 15235 catalog_manager.cc:1259] Loaded cluster ID: 1e961c0586f84bb496b21339f9406261
I20260927 21:09:39.164215 15235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: loading cluster ID for follower catalog manager: success
W20260927 21:09:39.164433 15235 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 13bc302be34c4c1c830e3717f4db5ae4: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260927 21:09:39.164486 15235 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260927 21:09:39.164527 15224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:39.164853 15224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:39.164951 15224 catalog_manager.cc:1259] Loaded cluster ID: 1e961c0586f84bb496b21339f9406261
I20260927 21:09:39.164984 15224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 21:09:39.165138 15224 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0d7409e9ad google::LogMessage::Fail() at ??:0
@ 0x7f0d740a188a google::LogMessage::SendToLog() at ??:0
@ 0x7f0d7409e4de google::LogMessage::Flush() at ??:0
@ 0x7f0d7409fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0d7bfbf0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0d7bfc88b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0d75f6c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0d75f61954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0d76dc56db start_thread at ??:0
@ 0x7f0d72b9c61f clone at ??:0
W20260927 21:09:39.266719 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:39.266906 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:39.266930 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:39.268049 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:39.268090 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:39.268115 15218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:09:39.269511 15218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32771
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:39.269809 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:39.270038 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:39.274884 15253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.274935 15218 server_base.cc:1034] running on GCE node
W20260927 21:09:39.274871 15252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:39.275162 15255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.275390 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:39.275661 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:39.276832 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790543379276817 us; error 32 us; skew 500 ppm
Sep 27 21:09:39 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:39.279289 15218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:09:39.279850 15218 webserver.cc:460] Webserver started at http://127.0.98.1:32771/ using document root <none> and password file <none>
I20260927 21:09:39.280090 15218 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:39.280157 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:39.281445 15218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:39.282145 15262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:39.282323 15218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:39.282419 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "5d62079001b342e29a3bee36cf17f8e4"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "8846f55977e424104807fd8291fc0315"
server_key_iv: "22ef6c7fd750d4b811722f679e6d9517"
server_key_version: "encryptionkey@0"
I20260927 21:09:39.282791 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:39.306131 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:39.308652 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:39.308830 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:39.309069 15218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:39.309523 15269 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:09:39.311481 15218 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:09:39.311539 15218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:09:39.311589 15218 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:09:39.312203 15218 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:09:39.312258 15218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:09:39.312323 15269 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap starting.
I20260927 21:09:39.319329 15218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43613
I20260927 21:09:39.319366 15376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43613 every 8 connection(s)
I20260927 21:09:39.319749 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:09:39.323164 15269 log.cc:824] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:39.324654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15218
I20260927 21:09:39.324919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44999
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38943
--webserver_port=43175
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:09:39.325733 15269 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:39.326026 15269 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Bootstrap complete.
I20260927 21:09:39.327277 15269 ts_tablet_manager.cc:1397] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260927 21:09:39.328758 15269 raft_consensus.cc:348] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:39.329087 15269 raft_consensus.cc:729] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d62079001b342e29a3bee36cf17f8e4, State: Initialized, Role: FOLLOWER
I20260927 21:09:39.329236 15269 consensus_queue.cc:260] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:39.329488 15269 ts_tablet_manager.cc:1428] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260927 21:09:39.376094 15379 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:09:39.320395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:49535)
0927 21:09:39.320578 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.320645 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:39.321128 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:39.375897 (+ 54769us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
W20260927 21:09:39.376384 15377 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38943 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38943: Client connection negotiation failed: client connection to 127.0.98.62:38943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:09:39.376756 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.376632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:35485)
0927 21:09:39.376685 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.376731 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 21:09:39.377067 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.376978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:49985)
0927 21:09:39.377023 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.377049 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:09:39.377183 15377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:09:39.406073 15381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:39.406265 15381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:39.406286 15381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:39.407476 15381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:39.407516 15381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:39.407567 15381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:09:39.408977 15381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43175
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:39.409200 15381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:39.409420 15381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:39.414168 15387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:39.414175 15390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:39.414176 15388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.414441 15381 server_base.cc:1034] running on GCE node
I20260927 21:09:39.414753 15381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:39.415002 15381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:39.416152 15381 hybrid_clock.cc:648] HybridClock initialized: now 1790543379416133 us; error 29 us; skew 500 ppm
Sep 27 21:09:39 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:39.418596 15381 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:09:39.419072 15381 webserver.cc:460] Webserver started at http://127.0.98.2:43175/ using document root <none> and password file <none>
I20260927 21:09:39.419343 15381 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:39.419412 15381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:39.420676 15381 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:09:39.421406 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:39.421582 15381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:09:39.421679 15381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "af25509c06c3451d9d87f48fdbded8c9"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "08ed2fbe5743d7e95a8308a7c2767525"
server_key_iv: "f69ca5fc488ba9555d6371aef7eada02"
server_key_version: "encryptionkey@0"
I20260927 21:09:39.422029 15381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:39.437769 15381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:39.440109 15381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:39.440274 15381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:39.440497 15381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:39.440922 15404 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:09:39.442317 15381 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:09:39.442396 15381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:09:39.442443 15381 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:09:39.443153 15381 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:09:39.443212 15381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:39.443265 15404 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Bootstrap starting.
I20260927 21:09:39.444733 15404 log.cc:824] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:39.446987 15404 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:39.447294 15404 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Bootstrap complete.
I20260927 21:09:39.447458 15404 ts_tablet_manager.cc:1397] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260927 21:09:39.448701 15404 raft_consensus.cc:348] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:39.448975 15404 raft_consensus.cc:729] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af25509c06c3451d9d87f48fdbded8c9, State: Initialized, Role: FOLLOWER
I20260927 21:09:39.449154 15404 consensus_queue.cc:260] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:39.449438 15404 ts_tablet_manager.cc:1428] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:09:39.450253 15381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44999
I20260927 21:09:39.450290 15513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44999 every 8 connection(s)
I20260927 21:09:39.450601 15381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:09:39.451565 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.451260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:49377)
0927 21:09:39.451395 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.451458 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
W20260927 21:09:39.451769 15514 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38943 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38943: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
I20260927 21:09:39.452046 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.451964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:60673)
0927 21:09:39.452007 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.452031 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:09:39.452268 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.452217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:51369)
0927 21:09:39.452236 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.452257 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 21:09:39.452368 15514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:09:39.459937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15381
I20260927 21:09:39.460193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38943
--webserver_port=44139
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:39.544056 15517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:39.544281 15517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:39.544318 15517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:39.545434 15517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:39.545500 15517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:39.545545 15517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:09:39.546960 15517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44139
--tserver_master_addrs=127.0.98.62:38943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:39.547226 15517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:39.547451 15517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:39.552099 15525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.552181 15517 server_base.cc:1034] running on GCE node
W20260927 21:09:39.552103 15523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:39.552112 15522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:39.552479 15517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:39.552663 15517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:39.553805 15517 hybrid_clock.cc:648] HybridClock initialized: now 1790543379553790 us; error 38 us; skew 500 ppm
Sep 27 21:09:39 dist-test-slave-8bgs krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:39.556232 15517 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:09:39.556680 15517 webserver.cc:460] Webserver started at http://127.0.98.3:44139/ using document root <none> and password file <none>
I20260927 21:09:39.556898 15517 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:39.556942 15517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:39.558215 15517 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:09:39.558768 15532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:39.558933 15517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:39.559013 15517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894"
format_stamp: "Formatted at 2026-09-27 21:09:37 on dist-test-slave-8bgs"
server_key: "bfeb76ef920579d2c10a051b6a7a2a0f"
server_key_iv: "253e6cdfcca6d994d1bb0811aa514678"
server_key_version: "encryptionkey@0"
I20260927 21:09:39.559381 15517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:39.580793 15517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:39.582620 15517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:39.582736 15517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:39.582913 15517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:39.583259 15517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:39.583294 15517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:39.583334 15517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:39.583362 15517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:39.588985 15517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39971
I20260927 21:09:39.589061 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39971 every 8 connection(s)
I20260927 21:09:39.589376 15517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:09:39.590416 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.590011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:43951)
0927 21:09:39.590168 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.590234 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260927 21:09:39.590819 15646 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38943 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38943: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
I20260927 21:09:39.591126 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.591003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:36833)
0927 21:09:39.591050 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.591074 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 21:09:39.591413 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:39.591327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:57015)
0927 21:09:39.591371 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:39.591393 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:09:39.591504 15646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:09:39.594485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15517
I20260927 21:09:40.377739 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.377520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:36549)
0927 21:09:40.377654 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:40.377689 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260927 21:09:40.452877 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.452686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:36211)
0927 21:09:40.452803 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:40.452839 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260927 21:09:40.591949 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.591763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:56421)
0927 21:09:40.591878 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:40.591906 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260927 21:09:40.635020 15653 raft_consensus.cc:482] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:40.635164 15653 raft_consensus.cc:504] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:40.635509 15653 leader_election.cc:288] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971), 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613)
Sep 27 21:09:40 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
Sep 27 21:09:40 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 21:09:40.646909 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.635831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48333 (local address 127.0.98.1:43613)
0927 21:09:40.635973 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:40.635979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:40.636004 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:09:40.636091 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:40.636093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:40.636119 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:40.636207 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:40.637288 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.638714 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.640235 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.640851 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.640965 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:40.643526 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:40.643530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:40.643530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:40.643542 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:40.645220 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:40.645657 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:40.645660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:40.645661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:40.645758 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:40.646013 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:40.646014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:40.646014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:40.646264 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:40.646368 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:40.646670 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:09:40.646794 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":54,"threads_started":1}
I20260927 21:09:40.646909 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.635666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43613 (local address 127.0.98.2:48333)
0927 21:09:40.635730 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:40.635760 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:40.635944 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:40.636266 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:40.636270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:40.636549 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:40.637145 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:40.637152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.638829 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.638830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:40.640146 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:40.640149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.640947 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.640948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:40.641092 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:40.641719 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:40.641729 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:40.643357 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:40.645334 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:40.645340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:40.645351 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:40.645570 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:40.645831 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:40.645832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:40.645833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:40.645944 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:40.646345 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:40.646351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:40.646487 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:09:40.646594 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:09:40.646772 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":69}
I20260927 21:09:40.646973 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.635789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54133 (local address 127.0.98.3:39971)
0927 21:09:40.636117 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:40.636125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:40.636155 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:09:40.636617 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:40.636620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:40.636654 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:40.636762 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:40.637292 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.638714 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.640235 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.640851 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.640965 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:40.643886 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:40.643890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:40.643891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:40.643902 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:40.645589 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:40.645961 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:40.645964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:40.645964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:40.646056 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:40.646253 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:40.646254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:40.646254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:40.646476 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:40.646561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:40.646742 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 21:09:40.646866 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":54,"threads_started":1}
I20260927 21:09:40.647396 15331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c566ee59e44ff7a506a8058dc0c73d" candidate_uuid: "af25509c06c3451d9d87f48fdbded8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5d62079001b342e29a3bee36cf17f8e4" is_pre_election: true
I20260927 21:09:40.647523 15331 raft_consensus.cc:2454] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af25509c06c3451d9d87f48fdbded8c9 in term 1.
I20260927 21:09:40.647703 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.635677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39971 (local address 127.0.98.2:54133)
0927 21:09:40.635781 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:40.635790 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:40.635945 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:40.636772 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:40.636773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:40.636962 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:40.637145 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:40.637152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.638820 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.638822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:40.640146 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:40.640149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:40.640948 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:40.640949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:40.641092 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:40.641773 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:40.641779 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:40.643768 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:40.645696 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:40.645697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:40.645698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:40.645872 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:40.646124 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:40.646124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:40.646125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:40.646188 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:40.646557 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:40.646558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:40.646603 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:09:40.646684 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:09:40.646837 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":129,"thread_start_us":51,"threads_started":1}
I20260927 21:09:40.647811 15398 leader_election.cc:302] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d62079001b342e29a3bee36cf17f8e4, af25509c06c3451d9d87f48fdbded8c9; no voters:
I20260927 21:09:40.647920 15653 raft_consensus.cc:2781] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:09:40.647986 15653 raft_consensus.cc:482] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:40.648013 15653 raft_consensus.cc:3037] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 1 FOLLOWER]: Advancing to term 2
W20260927 21:09:40.648427 15401 leader_election.cc:341] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971): Not found: Tablet not found: 66c566ee59e44ff7a506a8058dc0c73d
I20260927 21:09:40.649219 15653 raft_consensus.cc:504] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:40.649334 15653 leader_election.cc:288] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [CANDIDATE]: Term 2 election: Requested vote from peers 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971), 5d62079001b342e29a3bee36cf17f8e4 (127.0.98.1:43613)
I20260927 21:09:40.649570 15331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c566ee59e44ff7a506a8058dc0c73d" candidate_uuid: "af25509c06c3451d9d87f48fdbded8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5d62079001b342e29a3bee36cf17f8e4"
W20260927 21:09:40.649621 15401 leader_election.cc:341] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971): Not found: Tablet not found: 66c566ee59e44ff7a506a8058dc0c73d
I20260927 21:09:40.649657 15331 raft_consensus.cc:3037] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:09:40.650725 15331 raft_consensus.cc:2454] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af25509c06c3451d9d87f48fdbded8c9 in term 2.
I20260927 21:09:40.650935 15398 leader_election.cc:302] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d62079001b342e29a3bee36cf17f8e4, af25509c06c3451d9d87f48fdbded8c9; no voters: 11b4d640e05d4ddb8c0a2d9f9ea7f894
I20260927 21:09:40.651021 15653 raft_consensus.cc:2781] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:09:40.651171 15653 raft_consensus.cc:686] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [term 2 LEADER]: Becoming Leader. State: Replica: af25509c06c3451d9d87f48fdbded8c9, State: Running, Role: LEADER
I20260927 21:09:40.651296 15653 consensus_queue.cc:237] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:40.651487 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:40.651370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:50623)
0927 21:09:40.651418 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:40.651444 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:09:41.228003 15331 raft_consensus.cc:1261] T 66c566ee59e44ff7a506a8058dc0c73d P 5d62079001b342e29a3bee36cf17f8e4 [term 2 FOLLOWER]: Refusing update from remote peer af25509c06c3451d9d87f48fdbded8c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 21:09:41.228368 15659 consensus_queue.cc:1035] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 21:09:41.229336 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.229127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:52761)
0927 21:09:41.229238 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.229289 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":78,"threads_started":1}
W20260927 21:09:41.230139 15401 consensus_peers.cc:487] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9 -> Peer 11b4d640e05d4ddb8c0a2d9f9ea7f894 (127.0.98.3:39971): Couldn't send request to peer 11b4d640e05d4ddb8c0a2d9f9ea7f894. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66c566ee59e44ff7a506a8058dc0c73d. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:09:41.230165 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.230066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:49365)
0927 21:09:41.230103 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.230137 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 21:09:41.230194 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.229893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:50645)
0927 21:09:41.230128 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.230155 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":35,"threads_started":1}
I20260927 21:09:41.592691 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.592457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:57657)
0927 21:09:41.592600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.592634 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260927 21:09:41.764343 15669 ts_tablet_manager.cc:927] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Initiating tablet copy from peer af25509c06c3451d9d87f48fdbded8c9 (127.0.98.2:44999)
I20260927 21:09:41.764582 15669 tablet_copy_client.cc:316] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:44999
Sep 27 21:09:41 dist-test-slave-8bgs krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 21:09:41.771821 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.764869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44999 (local address 127.0.98.3:50689)
0927 21:09:41.764925 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.764948 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:41.765002 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:41.765212 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:41.765216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:41.765475 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:41.765541 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:41.765543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:41.766338 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:41.766339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:41.766890 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:41.766891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:41.767371 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:41.767372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:41.767429 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:41.767861 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:41.767865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:41.769236 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:41.770806 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:41.770812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:41.770814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:41.770998 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:41.771233 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:41.771234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:41.771234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:41.771288 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:41.771533 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:41.771539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:41.771583 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:09:41.771647 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:09:41.771734 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:09:41.771822 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.764958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50689 (local address 127.0.98.2:44999)
0927 21:09:41.765083 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:41.765087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:41.765095 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:09:41.765118 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:41.765119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:41.765130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:41.765196 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:41.765607 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:41.766267 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:41.766956 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:41.767311 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:41.767363 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:41.769332 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:41.769340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:41.769341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:41.769347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:41.770723 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:41.771067 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:41.771070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:41.771070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:41.771178 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:41.771342 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:41.771342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:41.771343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:41.771481 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:41.771541 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:41.771685 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:09:41.771757 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":40,"threads_started":1}
I20260927 21:09:41.772222 15487 tablet_copy_service.cc:140] P af25509c06c3451d9d87f48fdbded8c9: Received BeginTabletCopySession request for tablet 66c566ee59e44ff7a506a8058dc0c73d from peer 11b4d640e05d4ddb8c0a2d9f9ea7f894 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50689)
I20260927 21:09:41.772284 15487 tablet_copy_service.cc:161] P af25509c06c3451d9d87f48fdbded8c9: Beginning new tablet copy session on tablet 66c566ee59e44ff7a506a8058dc0c73d from peer 11b4d640e05d4ddb8c0a2d9f9ea7f894 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50689: session id = 11b4d640e05d4ddb8c0a2d9f9ea7f894-66c566ee59e44ff7a506a8058dc0c73d
I20260927 21:09:41.772753 15487 tablet_copy_source_session.cc:205] T 66c566ee59e44ff7a506a8058dc0c73d P af25509c06c3451d9d87f48fdbded8c9: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:09:41.773653 15669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c566ee59e44ff7a506a8058dc0c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:41.776368 15669 tablet_copy_client.cc:791] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: tablet copy: Starting download of 0 data blocks...
I20260927 21:09:41.776512 15669 tablet_copy_client.cc:655] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: tablet copy: Starting download of 1 WAL segments...
I20260927 21:09:41.777328 15669 tablet_copy_client.cc:531] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:09:41.777911 15669 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Bootstrap starting.
I20260927 21:09:41.779330 15669 log.cc:824] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:41.781517 15669 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:09:41.781687 15669 tablet_bootstrap.cc:492] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Bootstrap complete.
I20260927 21:09:41.781780 15669 ts_tablet_manager.cc:1397] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 21:09:41.782632 15669 raft_consensus.cc:348] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:41.782773 15669 raft_consensus.cc:729] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b4d640e05d4ddb8c0a2d9f9ea7f894, State: Initialized, Role: FOLLOWER
I20260927 21:09:41.782891 15669 consensus_queue.cc:260] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af25509c06c3451d9d87f48fdbded8c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44999 } } peers { permanent_uuid: "11b4d640e05d4ddb8c0a2d9f9ea7f894" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } } peers { permanent_uuid: "5d62079001b342e29a3bee36cf17f8e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43613 } }
I20260927 21:09:41.783185 15669 ts_tablet_manager.cc:1428] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894: Time spent starting tablet: real 0.001s user 0.001s sys 0.002s
I20260927 21:09:41.783433 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.783316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:50909)
0927 21:09:41.783371 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.783404 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:09:41.783509 15487 tablet_copy_service.cc:342] P af25509c06c3451d9d87f48fdbded8c9: Request end of tablet copy session 11b4d640e05d4ddb8c0a2d9f9ea7f894-66c566ee59e44ff7a506a8058dc0c73d received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50689
I20260927 21:09:41.783571 15487 tablet_copy_service.cc:433] P af25509c06c3451d9d87f48fdbded8c9: ending tablet copy session 11b4d640e05d4ddb8c0a2d9f9ea7f894-66c566ee59e44ff7a506a8058dc0c73d on tablet 66c566ee59e44ff7a506a8058dc0c73d with peer 11b4d640e05d4ddb8c0a2d9f9ea7f894
I20260927 21:09:41.783726 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:41.783643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:52481)
0927 21:09:41.783689 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:41.783711 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 21:09:42.230772 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:42.230541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:54465)
0927 21:09:42.230668 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:42.230707 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260927 21:09:42.230805 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:42.230581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:57023)
0927 21:09:42.230715 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:42.230768 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260927 21:09:42.271715 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:42.271582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:36961)
0927 21:09:42.271636 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:42.271678 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:09:42.271760 15600 raft_consensus.cc:1203] T 66c566ee59e44ff7a506a8058dc0c73d P 11b4d640e05d4ddb8c0a2d9f9ea7f894 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 21:09:42.272202 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:42.272105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:60849)
0927 21:09:42.272141 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:42.272170 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 21:09:43.231591 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:43.231377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:35455)
0927 21:09:43.231510 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:43.231549 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260927 21:09:43.272360 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:43.272159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:56927)
0927 21:09:43.272281 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:43.272316 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260927 21:09:43.272742 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:43.272554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:40373)
0927 21:09:43.272660 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:43.272705 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260927 21:09:44.232362 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:44.232130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:37727)
0927 21:09:44.232275 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:44.232314 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260927 21:09:44.273048 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:44.272793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:38475)
0927 21:09:44.272943 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:44.272989 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260927 21:09:44.273311 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:44.273095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:39695)
0927 21:09:44.273246 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:44.273273 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260927 21:09:45.233079 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:45.232884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:60443)
0927 21:09:45.233003 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:45.233033 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260927 21:09:45.273722 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:45.273500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:39419)
0927 21:09:45.273629 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:45.273678 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260927 21:09:45.274041 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:45.273857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:51391)
0927 21:09:45.273965 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:45.274003 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20260927 21:09:46.233767 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:46.233534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:40541)
0927 21:09:46.233681 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:46.233721 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":58,"threads_started":1}
I20260927 21:09:46.274482 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:46.274268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:35891)
0927 21:09:46.274377 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:46.274418 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260927 21:09:46.274641 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:46.274417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:51897)
0927 21:09:46.274533 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:46.274588 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260927 21:09:47.234465 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:47.234251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:32809)
0927 21:09:47.234376 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:47.234419 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260927 21:09:47.275206 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:47.274960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:37401)
0927 21:09:47.275103 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:47.275145 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260927 21:09:47.275337 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:47.275077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:49253)
0927 21:09:47.275229 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:47.275264 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260927 21:09:48.235145 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:48.234917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:37267)
0927 21:09:48.235047 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:48.235081 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260927 21:09:48.275882 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:48.275669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:48749)
0927 21:09:48.275808 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:48.275837 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260927 21:09:48.275942 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:48.275736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:51379)
0927 21:09:48.275869 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:48.275900 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 21:09:49.235917 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:49.235641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:38943)
0927 21:09:49.235795 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:49.235845 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260927 21:09:49.276615 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:49.276401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:55137)
0927 21:09:49.276532 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:49.276568 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260927 21:09:49.276635 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:49.276402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:51067)
0927 21:09:49.276565 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:49.276595 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260927 21:09:50.236716 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:50.236452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:55425)
0927 21:09:50.236617 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:50.236669 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260927 21:09:50.277366 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:50.277197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:51637)
0927 21:09:50.277299 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:50.277326 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":48,"threads_started":1}
I20260927 21:09:50.277366 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:50.277152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:45879)
0927 21:09:50.277278 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:50.277320 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260927 21:09:51.237322 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:51.237139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:33477)
0927 21:09:51.237246 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:51.237278 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260927 21:09:51.278035 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:51.277826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:37961)
0927 21:09:51.277959 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:51.277989 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260927 21:09:51.278043 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:51.277861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:47723)
0927 21:09:51.277975 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:51.278004 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":40,"threads_started":1}
I20260927 21:09:52.237995 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:52.237770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:40159)
0927 21:09:52.237911 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:52.237950 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260927 21:09:52.278705 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:52.278489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:36563)
0927 21:09:52.278619 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:52.278659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260927 21:09:52.278714 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:52.278508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:40079)
0927 21:09:52.278623 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:52.278659 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260927 21:09:53.238694 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:53.238486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:51287)
0927 21:09:53.238613 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:53.238649 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260927 21:09:53.279501 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:53.279278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:47673)
0927 21:09:53.279431 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:53.279461 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260927 21:09:53.279496 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:53.279278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:32769)
0927 21:09:53.279413 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:53.279449 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260927 21:09:54.239476 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:54.239248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.1:37703)
0927 21:09:54.239384 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:54.239432 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260927 21:09:54.280244 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:54.279999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.3:60213)
0927 21:09:54.280155 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:54.280197 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260927 21:09:54.280251 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:54.280016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38943 (local address 127.0.98.2:39873)
0927 21:09:54.280174 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:54.280210 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260927 21:09:54.602680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 15218
I20260927 21:09:54.607406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 15381
I20260927 21:09:54.612016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 15517
2026-09-27T21:09:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18403 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:09:54 dist-test-slave-8bgs krb5kdc[15718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:09:54 dist-test-slave-8bgs krb5kdc[15718](info): set up 2 sockets
Sep 27 21:09:54 dist-test-slave-8bgs krb5kdc[15718](info): commencing operation
krb5kdc: starting...
W20260927 21:09:55.639204 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:09:55 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:09:55.751987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:55.831609 15731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:55.831800 15731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:55.831821 15731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:55.832976 15731 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:09:55.833019 15731 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:09:55.833031 15731 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:09:55.833042 15731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:09:55.833062 15731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:55.833118 15731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:55.833135 15731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:09:55.833158 15731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:09:55.834625 15731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:55.834865 15731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:55.835040 15731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:55.839870 15739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:55.839887 15731 server_base.cc:1034] running on GCE node
W20260927 21:09:55.839854 15736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:55.839845 15737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:55.840312 15731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:55.840600 15731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:55.841774 15731 hybrid_clock.cc:648] HybridClock initialized: now 1790543395841759 us; error 34 us; skew 500 ppm
Sep 27 21:09:55 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:55.844188 15731 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:09:55.844636 15731 webserver.cc:460] Webserver started at http://127.0.98.62:36859/ using document root <none> and password file <none>
I20260927 21:09:55.844810 15731 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:55.844848 15731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:55.844933 15731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:55.845890 15731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "bdcc5c77d7734e78b0399002a877e60c"
format_stamp: "Formatted at 2026-09-27 21:09:55 on dist-test-slave-8bgs"
server_key: "44194dc78e0c2c96b54769c536226110"
server_key_iv: "6635fa7d6281436be37c8483fd69e886"
server_key_version: "encryptionkey@0"
I20260927 21:09:55.846187 15731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdcc5c77d7734e78b0399002a877e60c"
format_stamp: "Formatted at 2026-09-27 21:09:55 on dist-test-slave-8bgs"
server_key: "44194dc78e0c2c96b54769c536226110"
server_key_iv: "6635fa7d6281436be37c8483fd69e886"
server_key_version: "encryptionkey@0"
I20260927 21:09:55.847462 15731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:09:55.848142 15746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:55.848308 15731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:55.848412 15731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "bdcc5c77d7734e78b0399002a877e60c"
format_stamp: "Formatted at 2026-09-27 21:09:55 on dist-test-slave-8bgs"
server_key: "44194dc78e0c2c96b54769c536226110"
server_key_iv: "6635fa7d6281436be37c8483fd69e886"
server_key_version: "encryptionkey@0"
I20260927 21:09:55.848487 15731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:55.865542 15731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:55.867306 15731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:55.867473 15731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:55.871308 15731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46089
I20260927 21:09:55.871337 15798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46089 every 8 connection(s)
I20260927 21:09:55.871692 15731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:09:55.872490 15799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:55.874591 15799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap starting.
I20260927 21:09:55.875150 15799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:55.875399 15799 log.cc:824] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:55.876003 15799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: No bootstrap required, opened a new log
I20260927 21:09:55.876334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15731
I20260927 21:09:55.876431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:09:55.876636 392 external_mini_cluster.cc:1428] Setting key 6e3367eda42606bc9f6d43ef1c084b3a
I20260927 21:09:55.878479 15799 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:09:55.878656 15799 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:55.878695 15799 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcc5c77d7734e78b0399002a877e60c, State: Initialized, Role: FOLLOWER
I20260927 21:09:55.878821 15799 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:09:55.878908 15799 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:09:55.878942 15799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:09:55.878989 15799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:55.879535 15799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:09:55.879621 15799 leader_election.cc:302] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdcc5c77d7734e78b0399002a877e60c; no voters:
I20260927 21:09:55.879809 15799 leader_election.cc:288] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:09:55.879840 15804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:55.879947 15804 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 LEADER]: Becoming Leader. State: Replica: bdcc5c77d7734e78b0399002a877e60c, State: Running, Role: LEADER
I20260927 21:09:55.880033 15804 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:09:55.880059 15799 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:09:55.880336 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdcc5c77d7734e78b0399002a877e60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } } }
I20260927 21:09:55.880405 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:55.880528 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcc5c77d7734e78b0399002a877e60c. Latest consensus state: current_term: 1 leader_uuid: "bdcc5c77d7734e78b0399002a877e60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } } }
I20260927 21:09:55.880607 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: This master's current role is: LEADER
I20260927 21:09:55.881255 15808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:09:55.881749 15808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:09:55.883323 15808 catalog_manager.cc:1347] Generated new cluster ID: 8f7609923cd94894823040c2eaf9a851
I20260927 21:09:55.883378 15808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:09:55 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:55.887970 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:55.877124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.0.98.62:46089)
0927 21:09:55.877304 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:55.877312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:55.877335 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:09:55.877868 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:55.877874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:55.877919 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:55.878027 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:55.878942 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:55.880498 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:55.881612 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:55.882181 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:55.882292 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:55.884984 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:55.884988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:55.884988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:55.884998 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:55.886485 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:55.886853 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:55.886855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:55.886855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:55.886931 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:55.887144 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:55.887145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:55.887145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:55.887431 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:55.887595 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:55.887664 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:09:55.887787 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260927 21:09:55.892467 15808 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:09:55.892963 15808 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:09:55.898012 15808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Generated new TSK 0
I20260927 21:09:55.898180 15808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:09:55.927623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46089
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:56.007261 15825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:56.007452 15825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:56.007475 15825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:56.008651 15825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:56.008694 15825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:56.008720 15825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:09:56.010155 15825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:56.010404 15825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:56.010641 15825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:56.015599 15833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:56.015622 15830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:56.015591 15831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:56.015867 15825 server_base.cc:1034] running on GCE node
I20260927 21:09:56.016067 15825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:56.016335 15825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:56.017504 15825 hybrid_clock.cc:648] HybridClock initialized: now 1790543396017497 us; error 35 us; skew 500 ppm
Sep 27 21:09:56 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:56.019975 15825 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:09:56.020377 15825 webserver.cc:460] Webserver started at http://127.0.98.1:36335/ using document root <none> and password file <none>
I20260927 21:09:56.020614 15825 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:56.020684 15825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:56.020833 15825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:56.021837 15825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f6bd53f531b4610b0cc0b63e28f23c6"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "e726581ed4c746fcce303c98c08aca01"
server_key_iv: "0311cb76e4f2843c97003ca6fdf899a9"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.022197 15825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f6bd53f531b4610b0cc0b63e28f23c6"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "e726581ed4c746fcce303c98c08aca01"
server_key_iv: "0311cb76e4f2843c97003ca6fdf899a9"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.023562 15825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:09:56.024271 15840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.024435 15825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:09:56.024523 15825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "4f6bd53f531b4610b0cc0b63e28f23c6"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "e726581ed4c746fcce303c98c08aca01"
server_key_iv: "0311cb76e4f2843c97003ca6fdf899a9"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.024619 15825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:56.036876 15825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:56.038601 15825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:56.038793 15825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:56.039064 15825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:56.039420 15825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:56.039463 15825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.039497 15825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:56.039526 15825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.045439 15825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44837
I20260927 21:09:56.045540 15953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44837 every 8 connection(s)
I20260927 21:09:56.045830 15825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:09:56.052201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15825
I20260927 21:09:56.052310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
Sep 27 21:09:56 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:56.052534 392 external_mini_cluster.cc:1428] Setting key cd0c7234feed6cd6e41a16b2eaa0e02b
I20260927 21:09:56.056087 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.046508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:58045)
0927 21:09:56.046685 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.046719 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.047294 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.047556 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.047560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.048130 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:56.048316 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.048321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.049018 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.049019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.049821 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.049823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.050298 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.050300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.050472 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.051107 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:56.051117 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:56.052809 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:56.054658 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:56.054666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:56.054668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:56.054902 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:56.055178 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:56.055179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:56.055179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:56.055314 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:56.055599 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:56.055603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:56.055720 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.055802 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.055923 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260927 21:09:56.056105 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.046522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58045 (local address 127.0.98.62:46089)
0927 21:09:56.046563 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.046565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.046808 (+ 243us) server_negotiation.cc:407] Connection header received
0927 21:09:56.047398 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.047399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.047411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.047474 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:56.048417 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.048921 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.049926 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.050206 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.050287 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.052882 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:56.052884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:56.052884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:56.052892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:56.054543 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.054982 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.054983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.054984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.055055 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.055380 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.055380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.055381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.055533 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:56.055607 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.055849 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.055956 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:09:56.057034 15954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46089
I20260927 21:09:56.057144 15954 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:56.057317 15954 heartbeater.cc:511] Master 127.0.98.62:46089 requested a full tablet report, sending...
I20260927 21:09:56.057888 15763 ts_manager.cc:194] Registered new tserver with Master: 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837)
I20260927 21:09:56.058499 15763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58045
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:09:56.075939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46089
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:56.155967 15959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:56.156167 15959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:56.156193 15959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:56.157264 15959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:56.157298 15959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:56.157323 15959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:09:56.158646 15959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:56.158823 15959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:56.158983 15959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:56.163683 15965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:56.163689 15967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:56.163727 15964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:56.163801 15959 server_base.cc:1034] running on GCE node
I20260927 21:09:56.164062 15959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:56.164317 15959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:56.165468 15959 hybrid_clock.cc:648] HybridClock initialized: now 1790543396165457 us; error 27 us; skew 500 ppm
Sep 27 21:09:56 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:56.167968 15959 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:09:56.168422 15959 webserver.cc:460] Webserver started at http://127.0.98.2:34439/ using document root <none> and password file <none>
I20260927 21:09:56.168653 15959 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:56.168721 15959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:56.168870 15959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:56.169883 15959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "70bceee4b3464d49b25455d0330ca9a1"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "4c7d66afd255e7acaebbef70b66cdbea"
server_key_iv: "ca1a9e444a92db4a8222a44f401b03bd"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.170238 15959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "70bceee4b3464d49b25455d0330ca9a1"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "4c7d66afd255e7acaebbef70b66cdbea"
server_key_iv: "ca1a9e444a92db4a8222a44f401b03bd"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.171627 15959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:09:56.172313 15974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.172477 15959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.172590 15959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "70bceee4b3464d49b25455d0330ca9a1"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "4c7d66afd255e7acaebbef70b66cdbea"
server_key_iv: "ca1a9e444a92db4a8222a44f401b03bd"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.172664 15959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:56.195281 15959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:56.197065 15959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:56.197223 15959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:56.197444 15959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:56.197742 15959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:56.197795 15959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.197837 15959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:56.197865 15959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.203747 15959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39159
I20260927 21:09:56.203791 16087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39159 every 8 connection(s)
I20260927 21:09:56.204105 15959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:09:56 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:56.210354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 15959
I20260927 21:09:56.210469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:09:56.210722 392 external_mini_cluster.cc:1428] Setting key 66574c85f87fcd868491c55a9c46f1c0
I20260927 21:09:56.213340 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.204771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43205 (local address 127.0.98.62:46089)
0927 21:09:56.204852 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.204854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.205021 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:09:56.205478 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.205479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.205489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.205541 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:56.206429 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.206917 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.207994 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.208189 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.208246 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.210271 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:56.210273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:56.210273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:56.210280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:56.211868 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.212292 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.212293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.212294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.212353 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.212694 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.212695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.212696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.212825 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:56.212884 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.213208 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.213277 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:09:56.213627 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.204721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:43205)
0927 21:09:56.204895 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.204931 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.205397 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.205601 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.205605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.206161 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:56.206350 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.206354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.206996 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.206997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.207918 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.207921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.208263 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.208264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.208368 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.208813 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:56.208821 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:56.210173 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:56.211972 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:56.211978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:56.211980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:56.212202 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:56.212428 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:56.212429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:56.212430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:56.212603 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:56.212894 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:56.212901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:56.213068 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.213189 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.213365 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260927 21:09:56.214129 16088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46089
I20260927 21:09:56.214231 16088 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:56.214413 16088 heartbeater.cc:511] Master 127.0.98.62:46089 requested a full tablet report, sending...
I20260927 21:09:56.214769 15763 ts_manager.cc:194] Registered new tserver with Master: 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159)
I20260927 21:09:56.215260 15763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43205
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:09:56.240712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46089
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:09:56.319865 16093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:09:56.320102 16093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:09:56.320142 16093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:09:56.321285 16093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:09:56.321347 16093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:09:56.321393 16093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:09:56.322877 16093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:09:56.323164 16093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:09:56.323397 16093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:09:56.328195 16098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:09:56.328222 16101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:56.328374 16093 server_base.cc:1034] running on GCE node
W20260927 21:09:56.328222 16099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:09:56.328657 16093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:09:56.328896 16093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:09:56.330049 16093 hybrid_clock.cc:648] HybridClock initialized: now 1790543396330035 us; error 28 us; skew 500 ppm
Sep 27 21:09:56 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:09:56.332573 16093 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:09:56.333019 16093 webserver.cc:460] Webserver started at http://127.0.98.3:38663/ using document root <none> and password file <none>
I20260927 21:09:56.333252 16093 fs_manager.cc:359] Metadata directory not provided
I20260927 21:09:56.333299 16093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:09:56.333461 16093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:09:56.334476 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "bfabea23458a4306a84b14121a378304"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "f029c4b01788b376c3f18f07c0dd111e"
server_key_iv: "a4338a9959104a7132a07e0960a89bcc"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.334831 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bfabea23458a4306a84b14121a378304"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "f029c4b01788b376c3f18f07c0dd111e"
server_key_iv: "a4338a9959104a7132a07e0960a89bcc"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.336205 16093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:09:56.336910 16108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.337095 16093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.337182 16093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "bfabea23458a4306a84b14121a378304"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "f029c4b01788b376c3f18f07c0dd111e"
server_key_iv: "a4338a9959104a7132a07e0960a89bcc"
server_key_version: "encryptionkey@0"
I20260927 21:09:56.337250 16093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:09:56.349520 16093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:09:56.351613 16093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:09:56.351763 16093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:09:56.351994 16093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:09:56.352289 16093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:09:56.352322 16093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.352344 16093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:09:56.352388 16093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:09:56.358237 16093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45733
I20260927 21:09:56.358284 16221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45733 every 8 connection(s)
I20260927 21:09:56.358577 16093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:09:56 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:09:56.365552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16093
I20260927 21:09:56.365660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:09:56.365895 392 external_mini_cluster.cc:1428] Setting key da03ee9a3da2995ce9dba52deaf73b34
I20260927 21:09:56.367766 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.359308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35591 (local address 127.0.98.62:46089)
0927 21:09:56.359354 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.359356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.359595 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:09:56.360039 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.360040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.360051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.360102 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:56.360957 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.361448 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.362442 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.362643 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.362709 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.364830 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:56.364832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:56.364832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:56.364839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:56.366470 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.366925 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.366926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.366926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.366979 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.367260 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.367260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.367260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.367362 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:56.367414 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.367646 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.367704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:09:56.367959 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.359293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:35591)
0927 21:09:56.359452 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.359508 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.359962 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.360154 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.360157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.360699 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:09:56.360879 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.360884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.361547 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.361548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.362327 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.362329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.362728 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.362729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.362835 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.363390 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:09:56.363399 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:09:56.364746 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:09:56.366549 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:56.366556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:56.366559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:56.366857 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:56.367036 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:09:56.367037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:09:56.367037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:09:56.367204 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:09:56.367419 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:09:56.367423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:09:56.367534 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.367616 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.367729 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260927 21:09:56.368432 16222 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46089
I20260927 21:09:56.368511 16222 heartbeater.cc:464] Registering TS with master...
I20260927 21:09:56.368651 16222 heartbeater.cc:511] Master 127.0.98.62:46089 requested a full tablet report, sending...
I20260927 21:09:56.369002 15763 ts_manager.cc:194] Registered new tserver with Master: bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
I20260927 21:09:56.369377 15763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35591
I20260927 21:09:56.377537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:09:56.384686 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.378870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.0.98.62:46089)
0927 21:09:56.378924 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.378926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.379064 (+ 138us) server_negotiation.cc:407] Connection header received
0927 21:09:56.379138 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.379139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.379153 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.379206 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:09:56.379748 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.380296 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.381008 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.381191 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.381245 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.382419 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:09:56.382420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:09:56.382421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:09:56.382436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:09:56.383797 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.384103 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.384103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.384104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.384148 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:09:56.384294 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:09:56.384294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:09:56.384294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:09:56.384409 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:09:56.384476 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.384568 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.384627 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:09:56.386188 15763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:09:56.387821 15763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:09:56.394109 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.389915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44837 (local address 127.0.0.1:43986)
0927 21:09:56.390269 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.390277 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.390313 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.390523 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.390526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.390529 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.390656 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.390659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.392037 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.392038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.393094 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.393096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.393780 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.393877 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.393893 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.393959 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.394016 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":2,"thread_start_us":52,"threads_started":1}
I20260927 21:09:56.394181 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.390329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.0.98.2:39159)
0927 21:09:56.390478 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.390483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.390554 (+ 71us) server_negotiation.cc:407] Connection header received
0927 21:09:56.390637 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.390639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.390656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.390734 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.390952 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.392027 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393118 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.393836 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393914 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.393969 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.394047 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.394086 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260927 21:09:56.394255 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.389915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.0.1:43746)
0927 21:09:56.390069 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.390110 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.390159 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.390893 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.390893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.390895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.390936 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.390937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.391818 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.391819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.393101 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.393102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.394011 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.394013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.394079 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.394083 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.394133 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.394174 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260927 21:09:56.394435 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.390103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.0.98.1:44837)
0927 21:09:56.390293 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.390298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.390310 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:09:56.390391 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.390393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.390408 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.390516 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.390744 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.391924 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393197 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.393683 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393769 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.393831 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.394312 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.394363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260927 21:09:56.394515 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.390162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.0.98.3:45733)
0927 21:09:56.390696 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.390700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.390709 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:09:56.390742 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.390744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.390758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.390824 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.390988 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.391711 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393166 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.393837 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393921 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.393969 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.394390 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.394426 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":65,"threads_started":1}
I20260927 21:09:56.394111 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.389926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39159 (local address 127.0.0.1:35070)
0927 21:09:56.390500 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.390510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.390568 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.390813 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.390814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.390816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.390863 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.390864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.392111 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.392112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.393030 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.393032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.393917 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.393918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.393976 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.393979 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.394024 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.394056 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":45,"threads_started":1}
I20260927 21:09:56.395274 16156 tablet_service.cc:1467] Processing CreateTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 (DEFAULT_TABLE table=test-table [id=33ccdcf1f86a42bd873c3da292cebc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:56.395287 15888 tablet_service.cc:1467] Processing CreateTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 (DEFAULT_TABLE table=test-table [id=33ccdcf1f86a42bd873c3da292cebc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:56.395625 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f8fab809d549d389d3e108f9d5e5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:56.395628 15888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f8fab809d549d389d3e108f9d5e5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:56.395598 16022 tablet_service.cc:1467] Processing CreateTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 (DEFAULT_TABLE table=test-table [id=33ccdcf1f86a42bd873c3da292cebc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:09:56.395879 16022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3f8fab809d549d389d3e108f9d5e5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:09:56.398172 16236 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Bootstrap starting.
I20260927 21:09:56.398818 16238 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Bootstrap starting.
I20260927 21:09:56.398919 16236 tablet_bootstrap.cc:654] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:56.399531 16238 tablet_bootstrap.cc:654] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:56.399549 16236 log.cc:824] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:56.400089 16238 log.cc:824] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:56.400314 16236 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: No bootstrap required, opened a new log
I20260927 21:09:56.400389 16236 ts_tablet_manager.cc:1397] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:09:56.400439 16237 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap starting.
I20260927 21:09:56.401100 16237 tablet_bootstrap.cc:654] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Neither blocks nor log segments found. Creating new log.
I20260927 21:09:56.401491 16238 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: No bootstrap required, opened a new log
I20260927 21:09:56.401567 16238 ts_tablet_manager.cc:1397] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:09:56.402601 16236 raft_consensus.cc:348] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.402783 16236 raft_consensus.cc:374] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:56.402818 16236 raft_consensus.cc:729] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfabea23458a4306a84b14121a378304, State: Initialized, Role: FOLLOWER
I20260927 21:09:56.402938 16236 consensus_queue.cc:260] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.403220 16236 ts_tablet_manager.cc:1428] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:09:56.403359 16222 heartbeater.cc:503] Master 127.0.98.62:46089 was elected leader, sending a full tablet report...
I20260927 21:09:56.403541 16238 raft_consensus.cc:348] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.403664 16237 log.cc:824] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:09:56.403716 16238 raft_consensus.cc:374] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:56.403784 16238 raft_consensus.cc:729] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70bceee4b3464d49b25455d0330ca9a1, State: Initialized, Role: FOLLOWER
I20260927 21:09:56.403923 16238 consensus_queue.cc:260] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.404237 16238 ts_tablet_manager.cc:1428] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 21:09:56.404237 16088 heartbeater.cc:503] Master 127.0.98.62:46089 was elected leader, sending a full tablet report...
I20260927 21:09:56.404399 16237 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: No bootstrap required, opened a new log
I20260927 21:09:56.404466 16237 ts_tablet_manager.cc:1397] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:09:56.406035 16237 raft_consensus.cc:348] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.406198 16237 raft_consensus.cc:374] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:09:56.406243 16237 raft_consensus.cc:729] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6bd53f531b4610b0cc0b63e28f23c6, State: Initialized, Role: FOLLOWER
I20260927 21:09:56.406349 16237 consensus_queue.cc:260] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.406594 16237 ts_tablet_manager.cc:1428] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:09:56.406635 15954 heartbeater.cc:503] Master 127.0.98.62:46089 was elected leader, sending a full tablet report...
W20260927 21:09:56.454643 16089 tablet.cc:2369] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:09:56.546578 15955 tablet.cc:2369] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:09:56.609179 16223 tablet.cc:2369] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:09:56.749567 16244 raft_consensus.cc:482] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:56.749725 16244 raft_consensus.cc:504] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.749780 16243 raft_consensus.cc:482] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:09:56.749869 16243 raft_consensus.cc:504] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.750136 16244 leader_election.cc:288] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfabea23458a4306a84b14121a378304 (127.0.98.3:45733), 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159)
I20260927 21:09:56.750408 16243 leader_election.cc:288] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfabea23458a4306a84b14121a378304 (127.0.98.3:45733), 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837)
I20260927 21:09:56.754170 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51585 (local address 127.0.98.1:44837)
0927 21:09:56.750497 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.750499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.750631 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:09:56.750734 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.750735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.750825 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.750859 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.751120 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.751936 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753297 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.753819 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753862 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.753871 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.754069 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.754103 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:09:56.754169 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39555 (local address 127.0.98.2:39159)
0927 21:09:56.750844 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.750847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.750922 (+ 75us) server_negotiation.cc:407] Connection header received
0927 21:09:56.751010 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.751010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.751019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.751045 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.751271 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.751938 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753297 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.753818 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753862 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.753871 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.754080 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.754111 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:09:56.754319 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36111 (local address 127.0.98.3:45733)
0927 21:09:56.750497 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.750499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.750510 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:09:56.750655 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.750657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.750688 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.750722 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.751039 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.751817 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753298 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.753958 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.754026 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.754037 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.754223 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.754256 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":27}
I20260927 21:09:56.754522 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46823 (local address 127.0.98.3:45733)
0927 21:09:56.750837 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.750839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.750846 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:09:56.750864 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.750864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.750873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.751170 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:09:56.751198 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.751919 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753301 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.753979 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.754396 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.754409 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.754427 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.754462 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
I20260927 21:09:56.754308 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.98.1:46823)
0927 21:09:56.750638 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.750649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.750697 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.750999 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.751000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.751002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.751106 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.751109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.752000 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.752001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.753195 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.753198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.754081 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.754081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.754148 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.754152 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.754182 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.754227 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":66,"threads_started":1}
I20260927 21:09:56.754621 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.98.2:36111)
0927 21:09:56.750444 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.750456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.750516 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.750882 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.750883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.750885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.750962 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.750964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.751910 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.751910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.753187 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.753190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.754068 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.754070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.754139 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.754143 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.754463 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.754506 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260927 21:09:56.754416 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39159 (local address 127.0.98.1:39555)
0927 21:09:56.750870 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.750878 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.750943 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.751137 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.751138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.751140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.751196 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.751197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.752030 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.752031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.753199 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.753202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.753929 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.753986 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.753988 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.754253 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.754307 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260927 21:09:56.754772 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.750306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44837 (local address 127.0.98.2:51585)
0927 21:09:56.750548 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.750558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:09:56.750659 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:09:56.750934 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:09:56.750935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:09:56.750937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:09:56.751013 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.751016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.752030 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.752030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.753195 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:09:56.753197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.753929 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.753929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:09:56.753993 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.753997 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:09:56.754646 (+ 649us) client_negotiation.cc:240] Negotiation successful
0927 21:09:56.754691 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260927 21:09:56.754911 16176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "70bceee4b3464d49b25455d0330ca9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfabea23458a4306a84b14121a378304" is_pre_election: true
I20260927 21:09:56.755055 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfabea23458a4306a84b14121a378304" is_pre_election: true
I20260927 21:09:56.755100 16176 raft_consensus.cc:2454] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70bceee4b3464d49b25455d0330ca9a1 in term 0.
I20260927 21:09:56.755036 16042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70bceee4b3464d49b25455d0330ca9a1" is_pre_election: true
I20260927 21:09:56.755155 15908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "70bceee4b3464d49b25455d0330ca9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" is_pre_election: true
I20260927 21:09:56.755187 16042 raft_consensus.cc:2454] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in term 0.
I20260927 21:09:56.755242 15908 raft_consensus.cc:2454] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70bceee4b3464d49b25455d0330ca9a1 in term 0.
I20260927 21:09:56.755329 15978 leader_election.cc:302] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304; no voters:
I20260927 21:09:56.755393 15841 leader_election.cc:302] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6, 70bceee4b3464d49b25455d0330ca9a1; no voters:
I20260927 21:09:56.755456 16243 raft_consensus.cc:2781] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:09:56.755512 16243 raft_consensus.cc:482] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:56.755538 16243 raft_consensus.cc:3037] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:56.755556 16244 raft_consensus.cc:2781] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:09:56.755615 16244 raft_consensus.cc:482] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:09:56.755645 16244 raft_consensus.cc:3037] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:56.756278 16243 raft_consensus.cc:504] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.756294 16244 raft_consensus.cc:504] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.756415 16243 leader_election.cc:288] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [CANDIDATE]: Term 1 election: Requested vote from peers bfabea23458a4306a84b14121a378304 (127.0.98.3:45733), 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837)
I20260927 21:09:56.756430 16244 leader_election.cc:288] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 election: Requested vote from peers bfabea23458a4306a84b14121a378304 (127.0.98.3:45733), 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159)
I20260927 21:09:56.756537 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "70bceee4b3464d49b25455d0330ca9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfabea23458a4306a84b14121a378304"
I20260927 21:09:56.756609 16042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70bceee4b3464d49b25455d0330ca9a1"
I20260927 21:09:56.756625 16175 raft_consensus.cc:3037] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:09:56.756618 15908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "70bceee4b3464d49b25455d0330ca9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6"
I20260927 21:09:56.756681 16042 raft_consensus.cc:2379] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in current term 1: Already voted for candidate 70bceee4b3464d49b25455d0330ca9a1 in this term.
I20260927 21:09:56.756652 16176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3f8fab809d549d389d3e108f9d5e5e0" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfabea23458a4306a84b14121a378304"
I20260927 21:09:56.756696 15908 raft_consensus.cc:2379] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 70bceee4b3464d49b25455d0330ca9a1 in current term 1: Already voted for candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in this term.
I20260927 21:09:56.756912 15841 leader_election.cc:302] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6; no voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304
I20260927 21:09:56.757001 16244 raft_consensus.cc:2726] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 21:09:56.757408 16175 raft_consensus.cc:2454] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70bceee4b3464d49b25455d0330ca9a1 in term 1.
I20260927 21:09:56.757577 15978 leader_election.cc:302] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304; no voters: 4f6bd53f531b4610b0cc0b63e28f23c6
I20260927 21:09:56.757675 16243 raft_consensus.cc:2781] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:09:56.757814 16243 raft_consensus.cc:686] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 LEADER]: Becoming Leader. State: Replica: 70bceee4b3464d49b25455d0330ca9a1, State: Running, Role: LEADER
I20260927 21:09:56.757903 16243 consensus_queue.cc:237] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:09:56.758714 15762 catalog_manager.cc:5495] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "70bceee4b3464d49b25455d0330ca9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } health_report { overall_health: HEALTHY } } }
I20260927 21:09:56.833950 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.831836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.0.98.2:39159)
0927 21:09:56.831905 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:09:56.831907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:09:56.831914 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:09:56.831949 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:09:56.831949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:09:56.831967 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:09:56.832025 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:09:56.832148 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.832624 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.833325 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:09:56.833502 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:09:56.833550 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:09:56.833686 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:09:56.833785 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:09:56.833844 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:09:56.833875 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 21:09:56.833892 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:09:56.835886 15908 raft_consensus.cc:1261] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Refusing update from remote peer 70bceee4b3464d49b25455d0330ca9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:56.835918 16175 raft_consensus.cc:1261] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Refusing update from remote peer 70bceee4b3464d49b25455d0330ca9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:09:56.836161 16250 consensus_queue.cc:1035] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:56.836293 16243 consensus_queue.cc:1035] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:09:56.838711 16253 mvcc.cc:205] Tried to move back new op lower bound from 7334065753436733440 to 7334065753120915456. Current Snapshot: MvccSnapshot[applied={T|T < 7334065753436733440}]
I20260927 21:09:56.838836 16254 mvcc.cc:205] Tried to move back new op lower bound from 7334065753436733440 to 7334065753120915456. Current Snapshot: MvccSnapshot[applied={T|T < 7334065753436733440}]
I20260927 21:09:56.838989 16255 mvcc.cc:205] Tried to move back new op lower bound from 7334065753436733440 to 7334065753120915456. Current Snapshot: MvccSnapshot[applied={T|T < 7334065753436733440}]
I20260927 21:09:56.841130 15763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:56.841218 15763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:09:56.841889 15763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Sending DeleteTablet for 3 replicas of tablet b3f8fab809d549d389d3e108f9d5e5e0
I20260927 21:09:56.842180 16156 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43746
I20260927 21:09:56.842254 15888 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260927 21:09:56.842255 16022 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35070
I20260927 21:09:56.842398 16263 tablet_replica.cc:330] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: stopping tablet replica
I20260927 21:09:56.842500 16263 raft_consensus.cc:2229] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:56.842576 16265 tablet_replica.cc:330] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: stopping tablet replica
I20260927 21:09:56.842576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 15825
I20260927 21:09:56.842695 16265 raft_consensus.cc:2229] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:09:56.842639 16263 raft_consensus.cc:2258] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:56.843359 16264 tablet_replica.cc:330] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: stopping tablet replica
I20260927 21:09:56.843483 16264 raft_consensus.cc:2229] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:09:56.843518 16263 ts_tablet_manager.cc:1905] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:56.843623 16264 raft_consensus.cc:2258] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:09:56.844210 16264 ts_tablet_manager.cc:1905] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:09:56.845291 16263 ts_tablet_manager.cc:1918] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:56.845402 16263 log.cc:1196] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/b3f8fab809d549d389d3e108f9d5e5e0
I20260927 21:09:56.845698 16263 ts_tablet_manager.cc:1939] T b3f8fab809d549d389d3e108f9d5e5e0 P bfabea23458a4306a84b14121a378304: Deleting consensus metadata
I20260927 21:09:56.846119 15750 catalog_manager.cc:4841] TS bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): tablet b3f8fab809d549d389d3e108f9d5e5e0 (table test-table [id=33ccdcf1f86a42bd873c3da292cebc5e]) successfully deleted
I20260927 21:09:56.846186 16264 ts_tablet_manager.cc:1918] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:09:56.846246 16264 log.cc:1196] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/b3f8fab809d549d389d3e108f9d5e5e0
I20260927 21:09:56.846480 16264 ts_tablet_manager.cc:1939] T b3f8fab809d549d389d3e108f9d5e5e0 P 70bceee4b3464d49b25455d0330ca9a1: Deleting consensus metadata
I20260927 21:09:56.846837 15747 catalog_manager.cc:4841] TS 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): tablet b3f8fab809d549d389d3e108f9d5e5e0 (table test-table [id=33ccdcf1f86a42bd873c3da292cebc5e]) successfully deleted
W20260927 21:09:56.848891 15748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44837 (error 108)
I20260927 21:09:56.849097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 15959
I20260927 21:09:56.849998 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:09:56.849867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44837 (local address 127.0.0.1:43244)
0927 21:09:56.849929 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:09:56.849966 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:09:56.850136 15748 catalog_manager.cc:4584] TS 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3f8fab809d549d389d3e108f9d5e5e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44837: connect: Connection refused (error 111)
I20260927 21:09:56.853791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 16093
I20260927 21:09:56.858429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 15731
I20260927 21:10:16.863847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46089
--webserver_interface=127.0.98.62
--webserver_port=36859
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:16.944406 16267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:16.944607 16267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:16.944629 16267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:16.945772 16267 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:10:16.945813 16267 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:10:16.945827 16267 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:10:16.945838 16267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:10:16.945894 16267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:16.945912 16267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:16.945937 16267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:10:16.945966 16267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:10:16.947445 16267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:16.947708 16267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:16.947952 16267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:16.957181 16274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:16.957218 16273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:16.957235 16276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:16.957283 16267 server_base.cc:1034] running on GCE node
I20260927 21:10:16.957677 16267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:16.958016 16267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:16.959156 16267 hybrid_clock.cc:648] HybridClock initialized: now 1790543416959153 us; error 40 us; skew 500 ppm
Sep 27 21:10:16 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:16.965371 16267 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:10:16.965920 16267 webserver.cc:460] Webserver started at http://127.0.98.62:36859/ using document root <none> and password file <none>
I20260927 21:10:16.966183 16267 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:16.966272 16267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:16.967767 16267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:10:16.968575 16283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:16.968751 16267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:10:16.968865 16267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "bdcc5c77d7734e78b0399002a877e60c"
format_stamp: "Formatted at 2026-09-27 21:09:55 on dist-test-slave-8bgs"
server_key: "44194dc78e0c2c96b54769c536226110"
server_key_iv: "6635fa7d6281436be37c8483fd69e886"
server_key_version: "encryptionkey@0"
I20260927 21:10:16.969282 16267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:16.979966 16267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:16.981832 16267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:16.981997 16267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:16.986259 16267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46089
I20260927 21:10:16.986311 16335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46089 every 8 connection(s)
I20260927 21:10:16.986726 16267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:10:16.988300 16336 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:10:16.988759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16267
I20260927 21:10:16.989073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44837
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46089
--webserver_port=36335
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:10:16.989114 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap starting.
I20260927 21:10:16.997718 16336 log.cc:824] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:16.999982 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:10:17.000289 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap complete.
I20260927 21:10:17.002689 16336 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:17.002939 16336 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcc5c77d7734e78b0399002a877e60c, State: Initialized, Role: FOLLOWER
I20260927 21:10:17.003139 16336 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:17.003201 16336 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:10:17.003268 16336 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:10:17.003314 16336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:10:17.004364 16336 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:17.004451 16336 leader_election.cc:302] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdcc5c77d7734e78b0399002a877e60c; no voters:
I20260927 21:10:17.004670 16336 leader_election.cc:288] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:10:17.004705 16340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:10:17.004815 16340 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 LEADER]: Becoming Leader. State: Replica: bdcc5c77d7734e78b0399002a877e60c, State: Running, Role: LEADER
I20260927 21:10:17.004956 16336 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:10:17.005324 16340 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:17.005664 16341 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bdcc5c77d7734e78b0399002a877e60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } } }
I20260927 21:10:17.005753 16341 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:17.006016 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcc5c77d7734e78b0399002a877e60c. Latest consensus state: current_term: 2 leader_uuid: "bdcc5c77d7734e78b0399002a877e60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } } }
I20260927 21:10:17.006090 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:17.006193 16347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:10:17.007015 16347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:10:17.007201 16347 catalog_manager.cc:1259] Loaded cluster ID: 8f7609923cd94894823040c2eaf9a851
I20260927 21:10:17.007398 16347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:10:17.008538 16347 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:10:17.008713 16347 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Loaded TSK: 0
I20260927 21:10:17.021564 16347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Generated new TSK 1
I20260927 21:10:17.021672 16347 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:10:17.072412 16338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:17.072669 16338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:17.072710 16338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:17.073982 16338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:17.074049 16338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:17.074097 16338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:10:17.075660 16338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36335
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:17.075923 16338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:17.076155 16338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:17.082127 16364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:17.082156 16363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:17.082315 16338 server_base.cc:1034] running on GCE node
W20260927 21:10:17.082145 16366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:17.082608 16338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:17.082896 16338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:17.084050 16338 hybrid_clock.cc:648] HybridClock initialized: now 1790543417084014 us; error 50 us; skew 500 ppm
Sep 27 21:10:17 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:17.086674 16338 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:10:17.087255 16338 webserver.cc:460] Webserver started at http://127.0.98.1:36335/ using document root <none> and password file <none>
I20260927 21:10:17.087492 16338 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:17.087589 16338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:17.088967 16338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:10:17.089712 16373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.089919 16338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:17.090013 16338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "4f6bd53f531b4610b0cc0b63e28f23c6"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "e726581ed4c746fcce303c98c08aca01"
server_key_iv: "0311cb76e4f2843c97003ca6fdf899a9"
server_key_version: "encryptionkey@0"
I20260927 21:10:17.090391 16338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:17.116797 16338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:17.119042 16338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:17.119239 16338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:17.119482 16338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:17.119957 16380 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:10:17.121815 16338 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:10:17.121879 16338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:10:17.121922 16338 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:10:17.122566 16338 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:10:17.122627 16338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:10:17.122684 16380 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap starting.
I20260927 21:10:17.129944 16338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44837
I20260927 21:10:17.130028 16487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44837 every 8 connection(s)
I20260927 21:10:17.130470 16338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:10:17.133533 16380 log.cc:824] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:17.134716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16338
I20260927 21:10:17.135007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39159
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46089
--webserver_port=34439
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:10:17.136354 16380 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:10:17.136646 16380 tablet_bootstrap.cc:492] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap complete.
I20260927 21:10:17.137828 16380 ts_tablet_manager.cc:1397] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260927 21:10:17.139117 16380 raft_consensus.cc:348] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:10:17.139443 16380 raft_consensus.cc:729] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6bd53f531b4610b0cc0b63e28f23c6, State: Initialized, Role: FOLLOWER
Sep 27 21:10:17 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:17.140995 16380 consensus_queue.cc:260] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } } peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } }
I20260927 21:10:17.141266 16380 ts_tablet_manager.cc:1428] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260927 21:10:17.144867 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.131368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47433 (local address 127.0.98.62:46089)
0927 21:10:17.131555 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.131565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.133969 (+ 2404us) server_negotiation.cc:407] Connection header received
0927 21:10:17.134604 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.134612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.134672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.134823 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:17.135866 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.137076 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.138177 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.138790 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.138873 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.141369 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:17.141373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:17.141374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:17.141389 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:17.143134 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.143575 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.143578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.143578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.143672 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.143933 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.143934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.143934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.144164 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:17.144257 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.144564 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.144684 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260927 21:10:17.144867 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.131228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:47433)
0927 21:10:17.133759 (+ 2531us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.133810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.134352 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.134980 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.134985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.135435 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:17.135722 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.135729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.137182 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.137183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.138066 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.138069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.139005 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.139006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.139166 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.139781 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:17.139790 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:17.141234 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:17.143234 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:17.143241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:17.143243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:17.143466 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:17.143746 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:17.143747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:17.143747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:17.143860 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:17.144249 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:17.144255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:17.144390 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.144497 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.144675 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2479,"thread_start_us":94,"threads_started":1}
I20260927 21:10:17.145820 16488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46089
I20260927 21:10:17.145913 16488 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:17.146078 16488 heartbeater.cc:511] Master 127.0.98.62:46089 requested a full tablet report, sending...
I20260927 21:10:17.146775 16300 ts_manager.cc:194] Registered new tserver with Master: 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837)
I20260927 21:10:17.147915 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47433
I20260927 21:10:17.148746 16488 heartbeater.cc:503] Master 127.0.98.62:46089 was elected leader, sending a full tablet report...
I20260927 21:10:17.152856 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.147406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44837 (local address 127.0.0.1:60166)
0927 21:10:17.149208 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.149243 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.149311 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.149583 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.149586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.149588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:17.149637 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.149638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.150897 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.150897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.151760 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.151764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.152631 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.152632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.152694 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.152704 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.152765 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.152803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1774,"thread_start_us":80,"threads_started":1}
I20260927 21:10:17.152907 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.147630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.0.98.1:44837)
0927 21:10:17.149432 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.149436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.149445 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:10:17.149477 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.149479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.149492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.149564 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:17.149735 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.150810 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.151837 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.152541 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.152616 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.152667 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.152798 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.152834 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1773,"thread_start_us":58,"threads_started":1}
I20260927 21:10:17.153231 16422 tablet_service.cc:1514] Processing DeleteTablet for tablet b3f8fab809d549d389d3e108f9d5e5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:09:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60166
I20260927 21:10:17.153767 16497 tablet_replica.cc:330] stopping tablet replica
I20260927 21:10:17.153851 16497 raft_consensus.cc:2229] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:17.153935 16497 raft_consensus.cc:2258] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:17.154246 16497 ts_tablet_manager.cc:1905] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:17.156497 16497 ts_tablet_manager.cc:1918] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:17.156559 16497 log.cc:1196] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal/wals/b3f8fab809d549d389d3e108f9d5e5e0
I20260927 21:10:17.156783 16497 ts_tablet_manager.cc:1939] T b3f8fab809d549d389d3e108f9d5e5e0 P 4f6bd53f531b4610b0cc0b63e28f23c6: Deleting consensus metadata
I20260927 21:10:17.157233 16285 catalog_manager.cc:4841] TS 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837): tablet b3f8fab809d549d389d3e108f9d5e5e0 (table test-table [id=33ccdcf1f86a42bd873c3da292cebc5e]) successfully deleted
W20260927 21:10:17.224895 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:17.225135 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:17.225178 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:17.226315 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:17.226404 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:17.226451 16493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:10:17.227999 16493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34439
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:17.228271 16493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:17.228504 16493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:17.235131 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:17.235118 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:17.235352 16493 server_base.cc:1034] running on GCE node
W20260927 21:10:17.235582 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:17.235822 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:17.236127 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:17.237298 16493 hybrid_clock.cc:648] HybridClock initialized: now 1790543417237274 us; error 38 us; skew 500 ppm
Sep 27 21:10:17 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:17.239811 16493 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:10:17.240314 16493 webserver.cc:460] Webserver started at http://127.0.98.2:34439/ using document root <none> and password file <none>
I20260927 21:10:17.240559 16493 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:17.240649 16493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:17.242089 16493 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:10:17.242836 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.242988 16493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:10:17.243131 16493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "70bceee4b3464d49b25455d0330ca9a1"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "4c7d66afd255e7acaebbef70b66cdbea"
server_key_iv: "ca1a9e444a92db4a8222a44f401b03bd"
server_key_version: "encryptionkey@0"
I20260927 21:10:17.243476 16493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:17.265482 16493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:17.268030 16493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:17.268201 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:17.268451 16493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:17.268798 16493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:17.268832 16493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.268854 16493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:17.268904 16493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.274914 16493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39159
I20260927 21:10:17.274964 16625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39159 every 8 connection(s)
I20260927 21:10:17.275307 16493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:10:17 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:17.282907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16493
I20260927 21:10:17.283236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45733
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46089
--webserver_port=38663
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:10:17.284401 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.275989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34437 (local address 127.0.98.62:46089)
0927 21:10:17.276074 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.276077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.276288 (+ 211us) server_negotiation.cc:407] Connection header received
0927 21:10:17.276763 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.276764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.276778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.276827 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:17.277813 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.278304 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.279413 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.279619 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.279673 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.281634 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:17.281635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:17.281635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:17.281641 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:17.282931 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.283433 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.283434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.283434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.283499 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.283737 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.283738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.283739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.283862 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:17.283951 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.284215 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.284302 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:10:17.284585 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.275950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:34437)
0927 21:10:17.276154 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.276195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.276674 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.276889 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.276892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.277451 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:17.277721 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.277726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.278391 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.278391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.279338 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.279341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.279699 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.279700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.279803 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.280218 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:17.280226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:17.281552 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:17.283064 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:17.283070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:17.283073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:17.283318 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:17.283559 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:17.283560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:17.283560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:17.283664 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:17.283942 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:17.283948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:17.284075 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.284181 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.284338 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":118,"threads_started":1}
I20260927 21:10:17.285236 16626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46089
I20260927 21:10:17.285331 16626 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:17.285504 16626 heartbeater.cc:511] Master 127.0.98.62:46089 requested a full tablet report, sending...
I20260927 21:10:17.285912 16300 ts_manager.cc:194] Registered new tserver with Master: 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159)
I20260927 21:10:17.286283 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34437
W20260927 21:10:17.364956 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:17.365159 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:17.365181 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:17.366348 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:17.366390 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:17.366452 16629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:10:17.368014 16629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38663
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:17.368266 16629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:17.368508 16629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:17.373508 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:17.373548 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:17.373514 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:17.374231 16629 server_base.cc:1034] running on GCE node
I20260927 21:10:17.374439 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:17.374687 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:17.375849 16629 hybrid_clock.cc:648] HybridClock initialized: now 1790543417375834 us; error 30 us; skew 500 ppm
Sep 27 21:10:17 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:17.378253 16629 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:10:17.378669 16629 webserver.cc:460] Webserver started at http://127.0.98.3:38663/ using document root <none> and password file <none>
I20260927 21:10:17.378903 16629 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:17.378981 16629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:17.380302 16629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:17.380903 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.381181 16629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:17.381289 16629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "bfabea23458a4306a84b14121a378304"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "f029c4b01788b376c3f18f07c0dd111e"
server_key_iv: "a4338a9959104a7132a07e0960a89bcc"
server_key_version: "encryptionkey@0"
I20260927 21:10:17.381641 16629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:17.407984 16629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:17.409857 16629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:17.410039 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:17.410298 16629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:17.410629 16629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:17.410687 16629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.410727 16629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:17.410758 16629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.416455 16629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45733
I20260927 21:10:17.416512 16757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45733 every 8 connection(s)
I20260927 21:10:17.416898 16629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:10:17.418567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16629
Sep 27 21:10:17 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:17.426631 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.419017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.0.98.62:46089)
0927 21:10:17.419167 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.419169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.419287 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:10:17.419349 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.419349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.419359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.419415 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:17.420134 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.420854 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.422066 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.422261 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.422381 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.423884 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:17.423886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:17.423886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:17.423892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:17.425479 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.425845 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.425846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.425846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.425905 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.426117 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.426118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.426118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.426233 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:17.426409 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.426445 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.426522 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260927 21:10:17.428431 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.417710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44991 (local address 127.0.98.62:46089)
0927 21:10:17.417760 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.417762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.418001 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:10:17.418542 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.418544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.418557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.418606 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:17.420008 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.420895 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.422330 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.422627 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.422691 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.425458 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:17.425459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:17.425459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:17.425466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:17.427019 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.427462 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.427463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.427463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.427518 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.427791 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.427791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.427792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.427909 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:17.427988 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.428265 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.428342 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:10:17.428627 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.417696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:44991)
0927 21:10:17.417870 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.417907 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.418425 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.418678 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.418683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.419593 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:17.419901 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.419908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.421032 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.421033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.422240 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.422242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.422962 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.422964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.423196 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.423790 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:17.423798 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:17.425357 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:17.427118 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:17.427124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:17.427127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:17.427364 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:17.427598 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:17.427599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:17.427599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:17.427714 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:17.427980 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:17.427986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:17.428127 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.428233 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.428401 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260927 21:10:17.429044 16758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46089
I20260927 21:10:17.429134 16758 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:17.429280 16758 heartbeater.cc:511] Master 127.0.98.62:46089 requested a full tablet report, sending...
I20260927 21:10:17.429608 16300 ts_manager.cc:194] Registered new tserver with Master: bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
I20260927 21:10:17.429937 16300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44991
I20260927 21:10:17.438185 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:10:17.440891 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.438565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.98.62:46089)
0927 21:10:17.438614 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.438615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.438726 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:10:17.438780 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.438781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.438814 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.438886 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:10:17.438964 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.439478 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.440219 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.440403 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.440467 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.440675 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:10:17.440719 (+ 44us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 21:10:17.440844 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51108: token expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 21:10:17.440970 16491 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51108: token expired
W20260927 21:10:17.441164 16225 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46089)
I20260927 21:10:17.449525 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.441416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.98.62:46089)
0927 21:10:17.441457 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.441459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.441467 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:10:17.441561 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.441562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.441572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.441633 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:17.442353 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.443161 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.444592 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.444864 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.444934 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.446332 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:17.446333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:17.446334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:17.446343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:17.448238 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.448721 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.448723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.448723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.448778 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:17.448999 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:17.449000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:17.449000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:17.449139 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:17.449248 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.449355 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.449429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:10:17.450577 16300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:10:17.452333 16300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:10:17.454427 16422 tablet_service.cc:1467] Processing CreateTablet for tablet 973c54646ce44a5f8cba0d62f91408b7 (DEFAULT_TABLE table=test-table [id=eacf119b60be417cae1d90962ec35df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:17.454613 16422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973c54646ce44a5f8cba0d62f91408b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:17.456089 16380 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap starting.
I20260927 21:10:17.456449 16380 tablet_bootstrap.cc:654] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:17.457049 16380 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: No bootstrap required, opened a new log
I20260927 21:10:17.457096 16380 ts_tablet_manager.cc:1397] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:17.457207 16380 raft_consensus.cc:348] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.457265 16380 raft_consensus.cc:374] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:17.457305 16380 raft_consensus.cc:729] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6bd53f531b4610b0cc0b63e28f23c6, State: Initialized, Role: FOLLOWER
I20260927 21:10:17.457343 16380 consensus_queue.cc:260] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.457415 16380 ts_tablet_manager.cc:1428] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:17.458458 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.454217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.0.1:59796)
0927 21:10:17.454365 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.454375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.454413 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.454822 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.454823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.454826 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:17.454870 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.454871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.456343 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.456344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.457263 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.457266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.458216 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.458217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.458280 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.458284 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.458333 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.458379 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":35,"thread_start_us":102,"threads_started":1}
I20260927 21:10:17.458554 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.454257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.2:39159)
0927 21:10:17.454694 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.454697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.454706 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:10:17.454740 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.454743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.454759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.454976 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:17.455064 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.456238 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.457633 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.458187 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.458264 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.458401 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.458426 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.458468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":94,"threads_started":1}
I20260927 21:10:17.458904 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.454457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.0.98.3:45733)
0927 21:10:17.454639 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.454645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.454656 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:10:17.454699 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.454702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.454720 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.454811 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:17.454936 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.456229 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.457372 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.458125 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.458196 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.458255 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.458813 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.458846 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260927 21:10:17.458581 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.454141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39159 (local address 127.0.0.1:58976)
0927 21:10:17.454190 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.454201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.454238 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.454861 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.454862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.454863 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:17.454903 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.454905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.456325 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.456326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.457468 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.457470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.458287 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.458288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.458353 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.458356 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.458398 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.458448 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":24}
I20260927 21:10:17.459656 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 973c54646ce44a5f8cba0d62f91408b7 (DEFAULT_TABLE table=test-table [id=eacf119b60be417cae1d90962ec35df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:17.459910 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973c54646ce44a5f8cba0d62f91408b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:17.459805 16560 tablet_service.cc:1467] Processing CreateTablet for tablet 973c54646ce44a5f8cba0d62f91408b7 (DEFAULT_TABLE table=test-table [id=eacf119b60be417cae1d90962ec35df3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:17.460129 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973c54646ce44a5f8cba0d62f91408b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:17.462883 16768 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Bootstrap starting.
I20260927 21:10:17.463209 16769 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Bootstrap starting.
I20260927 21:10:17.463572 16768 tablet_bootstrap.cc:654] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:17.463913 16769 tablet_bootstrap.cc:654] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:17.464215 16768 log.cc:824] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:17.464453 16769 log.cc:824] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:17.465072 16768 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: No bootstrap required, opened a new log
I20260927 21:10:17.465137 16769 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: No bootstrap required, opened a new log
I20260927 21:10:17.465150 16768 ts_tablet_manager.cc:1397] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:10:17.465188 16769 ts_tablet_manager.cc:1397] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:10:17.467125 16768 raft_consensus.cc:348] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.467078 16769 raft_consensus.cc:348] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.467293 16768 raft_consensus.cc:374] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:17.467293 16769 raft_consensus.cc:374] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:17.467330 16769 raft_consensus.cc:729] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfabea23458a4306a84b14121a378304, State: Initialized, Role: FOLLOWER
I20260927 21:10:17.467330 16768 raft_consensus.cc:729] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70bceee4b3464d49b25455d0330ca9a1, State: Initialized, Role: FOLLOWER
I20260927 21:10:17.467446 16769 consensus_queue.cc:260] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.467478 16768 consensus_queue.cc:260] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.467730 16769 ts_tablet_manager.cc:1428] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:10:17.467730 16768 ts_tablet_manager.cc:1428] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:10:17.467773 16626 heartbeater.cc:503] Master 127.0.98.62:46089 was elected leader, sending a full tablet report...
I20260927 21:10:17.467861 16758 heartbeater.cc:503] Master 127.0.98.62:46089 was elected leader, sending a full tablet report...
W20260927 21:10:17.525923 16627 tablet.cc:2369] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:10:17.631608 16489 tablet.cc:2369] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:10:17.667461 16759 tablet.cc:2369] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:10:17.750483 16494 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:17.750638 16494 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.751066 16494 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
I20260927 21:10:17.755294 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.751319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39159 (local address 127.0.98.1:49857)
0927 21:10:17.751465 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.751474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.751540 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.751705 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.751706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.751709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:17.751792 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.751795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.752805 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.752807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.754128 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.754131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.755005 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.755006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.755074 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.755079 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.755161 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.755215 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":33,"thread_start_us":59,"threads_started":1}
I20260927 21:10:17.755301 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.751396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51147 (local address 127.0.98.3:45733)
0927 21:10:17.751459 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.751462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.751556 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:10:17.751599 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.751601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.751615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.751685 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:17.751892 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.752716 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.754249 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.754922 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.754992 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.755003 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.755201 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.755244 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:10:17.755303 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.751402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49857 (local address 127.0.98.2:39159)
0927 21:10:17.751466 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.751469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.751587 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:10:17.751614 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.751615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.751630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.751685 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:17.751893 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.752715 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.754249 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.754886 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.754978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.754991 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.755201 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.755252 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:10:17.755507 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.751306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.98.1:51147)
0927 21:10:17.751449 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.751462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:17.751540 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:17.751705 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:17.751706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:17.751708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:17.751799 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.751801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.752808 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.752808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.754128 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:17.754131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.755003 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.755005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:17.755079 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.755110 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:10:17.755381 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 21:10:17.755423 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":116,"thread_start_us":65,"threads_started":1}
I20260927 21:10:17.755712 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c54646ce44a5f8cba0d62f91408b7" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70bceee4b3464d49b25455d0330ca9a1" is_pre_election: true
I20260927 21:10:17.755908 16580 raft_consensus.cc:2454] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in term 0.
I20260927 21:10:17.755884 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c54646ce44a5f8cba0d62f91408b7" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfabea23458a4306a84b14121a378304" is_pre_election: true
I20260927 21:10:17.756039 16712 raft_consensus.cc:2454] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in term 0.
I20260927 21:10:17.756163 16374 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6, 70bceee4b3464d49b25455d0330ca9a1; no voters:
I20260927 21:10:17.756332 16494 raft_consensus.cc:2781] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:10:17.756405 16494 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:10:17.756433 16494 raft_consensus.cc:3037] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:17.757083 16494 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.757241 16494 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 election: Requested vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
I20260927 21:10:17.757375 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c54646ce44a5f8cba0d62f91408b7" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfabea23458a4306a84b14121a378304"
I20260927 21:10:17.757400 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973c54646ce44a5f8cba0d62f91408b7" candidate_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70bceee4b3464d49b25455d0330ca9a1"
I20260927 21:10:17.757452 16712 raft_consensus.cc:3037] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:17.757485 16580 raft_consensus.cc:3037] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:17.758141 16712 raft_consensus.cc:2454] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in term 1.
I20260927 21:10:17.758282 16580 raft_consensus.cc:2454] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6bd53f531b4610b0cc0b63e28f23c6 in term 1.
I20260927 21:10:17.758303 16377 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6, bfabea23458a4306a84b14121a378304; no voters:
I20260927 21:10:17.758415 16494 raft_consensus.cc:2781] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:17.758534 16494 raft_consensus.cc:686] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4f6bd53f531b4610b0cc0b63e28f23c6, State: Running, Role: LEADER
I20260927 21:10:17.758622 16494 consensus_queue.cc:237] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:17.759392 16299 catalog_manager.cc:5495] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } health_report { overall_health: HEALTHY } } }
I20260927 21:10:17.812237 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:17.809164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.0.98.1:44837)
0927 21:10:17.809334 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:17.809335 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:17.809342 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:10:17.809381 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:17.809381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:17.809400 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:17.809496 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:10:17.809666 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.810159 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.810899 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:17.811133 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:17.811217 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:17.811404 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:10:17.811965 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:10:17.812084 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:17.812103 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 21:10:17.812122 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260927 21:10:17.813777 16580 raft_consensus.cc:1261] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Refusing update from remote peer 4f6bd53f531b4610b0cc0b63e28f23c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:17.813802 16712 raft_consensus.cc:1261] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Refusing update from remote peer 4f6bd53f531b4610b0cc0b63e28f23c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:17.814141 16494 consensus_queue.cc:1035] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:17.814276 16776 consensus_queue.cc:1035] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:17.816975 16782 mvcc.cc:205] Tried to move back new op lower bound from 7334065839362748416 to 7334065839139819520. Current Snapshot: MvccSnapshot[applied={T|T < 7334065839362748416}]
I20260927 21:10:17.817265 16781 mvcc.cc:205] Tried to move back new op lower bound from 7334065839362748416 to 7334065839139819520. Current Snapshot: MvccSnapshot[applied={T|T < 7334065839362748416}]
I20260927 21:10:17.817406 16780 mvcc.cc:205] Tried to move back new op lower bound from 7334065839362748416 to 7334065839139819520. Current Snapshot: MvccSnapshot[applied={T|T < 7334065839362748416}]
I20260927 21:10:17.819527 16300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:17.819619 16300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:17.820319 16300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Sending DeleteTablet for 3 replicas of tablet 973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:17.820600 16422 tablet_service.cc:1514] Processing DeleteTablet for tablet 973c54646ce44a5f8cba0d62f91408b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60166
I20260927 21:10:17.820607 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet 973c54646ce44a5f8cba0d62f91408b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58976
I20260927 21:10:17.820762 16692 tablet_service.cc:1514] Processing DeleteTablet for tablet 973c54646ce44a5f8cba0d62f91408b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59796
I20260927 21:10:17.820801 16791 tablet_replica.cc:330] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: stopping tablet replica
I20260927 21:10:17.820811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 16338
I20260927 21:10:17.820880 16791 raft_consensus.cc:2229] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:17.820978 16791 raft_consensus.cc:2258] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:17.822191 16791 ts_tablet_manager.cc:1905] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:17.823948 16791 ts_tablet_manager.cc:1918] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:17.824038 16791 log.cc:1196] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:17.824281 16791 ts_tablet_manager.cc:1939] T 973c54646ce44a5f8cba0d62f91408b7 P 70bceee4b3464d49b25455d0330ca9a1: Deleting consensus metadata
I20260927 21:10:17.824704 16284 catalog_manager.cc:4841] TS 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): tablet 973c54646ce44a5f8cba0d62f91408b7 (table test-table [id=eacf119b60be417cae1d90962ec35df3]) successfully deleted
W20260927 21:10:17.826869 16285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44837 (error 108)
I20260927 21:10:17.827211 16792 tablet_replica.cc:330] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: stopping tablet replica
I20260927 21:10:17.827334 16792 raft_consensus.cc:2229] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 21:10:17.827441 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:10:17.827300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44837 (local address 127.0.0.1:60186)
0927 21:10:17.827347 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:17.827404 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44837: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:10:17.827457 16792 raft_consensus.cc:2258] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:17.827595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 16493
W20260927 21:10:17.827570 16285 catalog_manager.cc:4584] TS 4f6bd53f531b4610b0cc0b63e28f23c6 (127.0.98.1:44837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 973c54646ce44a5f8cba0d62f91408b7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44837: connect: Connection reset by peer (error 104)
I20260927 21:10:17.829221 16792 ts_tablet_manager.cc:1905] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:17.831243 16792 ts_tablet_manager.cc:1918] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:17.831321 16792 log.cc:1196] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:17.831604 16792 ts_tablet_manager.cc:1939] T 973c54646ce44a5f8cba0d62f91408b7 P bfabea23458a4306a84b14121a378304: Deleting consensus metadata
I20260927 21:10:17.832049 16287 catalog_manager.cc:4841] TS bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): tablet 973c54646ce44a5f8cba0d62f91408b7 (table test-table [id=eacf119b60be417cae1d90962ec35df3]) successfully deleted
I20260927 21:10:17.833573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 16629
I20260927 21:10:17.838989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 16267
I20260927 21:10:37.845028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46089
--webserver_interface=127.0.98.62
--webserver_port=36859
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:37.926733 16794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:37.926934 16794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:37.926957 16794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:37.928678 16794 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:10:37.928737 16794 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:10:37.928756 16794 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:10:37.928778 16794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:10:37.928808 16794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:37.928839 16794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:37.928860 16794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:10:37.928880 16794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:10:37.930446 16794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:37.930649 16794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:37.930811 16794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:37.935774 16799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:37.935804 16800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:37.935979 16794 server_base.cc:1034] running on GCE node
W20260927 21:10:37.935806 16802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:37.936336 16794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:37.936636 16794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:37.937811 16794 hybrid_clock.cc:648] HybridClock initialized: now 1790543437937800 us; error 27 us; skew 500 ppm
Sep 27 21:10:37 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:37.940231 16794 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:10:37.940646 16794 webserver.cc:460] Webserver started at http://127.0.98.62:36859/ using document root <none> and password file <none>
I20260927 21:10:37.940884 16794 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:37.940953 16794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:37.942333 16794 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:37.943063 16809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:37.943305 16794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:37.943401 16794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "bdcc5c77d7734e78b0399002a877e60c"
format_stamp: "Formatted at 2026-09-27 21:09:55 on dist-test-slave-8bgs"
server_key: "44194dc78e0c2c96b54769c536226110"
server_key_iv: "6635fa7d6281436be37c8483fd69e886"
server_key_version: "encryptionkey@0"
I20260927 21:10:37.943776 16794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:37.974910 16794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:37.976821 16794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:37.976985 16794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:37.981127 16794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46089
I20260927 21:10:37.981165 16861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46089 every 8 connection(s)
I20260927 21:10:37.981570 16794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:10:37.983007 16862 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:10:37.983680 16862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap starting.
I20260927 21:10:37.990373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16794
I20260927 21:10:37.990664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44837
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46089
--webserver_port=36335
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:10:37.992024 16862 log.cc:824] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:37.994240 16862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:10:37.994534 16862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Bootstrap complete.
I20260927 21:10:37.996773 16862 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:37.997012 16862 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcc5c77d7734e78b0399002a877e60c, State: Initialized, Role: FOLLOWER
I20260927 21:10:37.997222 16862 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:37.997323 16862 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:10:37.997380 16862 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:10:37.997429 16862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:10:37.998416 16862 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:37.998538 16862 leader_election.cc:302] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdcc5c77d7734e78b0399002a877e60c; no voters:
I20260927 21:10:37.998742 16862 leader_election.cc:288] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:10:37.998780 16866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:10:37.998894 16866 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [term 3 LEADER]: Becoming Leader. State: Replica: bdcc5c77d7734e78b0399002a877e60c, State: Running, Role: LEADER
I20260927 21:10:37.999024 16862 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:10:37.999035 16866 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } }
I20260927 21:10:37.999418 16867 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bdcc5c77d7734e78b0399002a877e60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } } }
I20260927 21:10:37.999434 16868 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcc5c77d7734e78b0399002a877e60c. Latest consensus state: current_term: 3 leader_uuid: "bdcc5c77d7734e78b0399002a877e60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcc5c77d7734e78b0399002a877e60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46089 } } }
I20260927 21:10:37.999513 16867 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:37.999541 16868 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:37.999951 16872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:10:38.000852 16872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:10:38.000958 16872 catalog_manager.cc:1259] Loaded cluster ID: 8f7609923cd94894823040c2eaf9a851
I20260927 21:10:38.001039 16872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:10:38.002070 16872 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:10:38.002271 16872 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Loaded TSK: 0
I20260927 21:10:38.002327 16872 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bdcc5c77d7734e78b0399002a877e60c: Loaded TSK: 1
F20260927 21:10:38.002552 16872 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f16bbbc89ad google::LogMessage::Fail() at ??:0
@ 0x7f16bbbcb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f16bbbc84de google::LogMessage::Flush() at ??:0
@ 0x7f16bbbc9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f16be335104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f16be332d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f16c3b221cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f16c3b2248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f16c3b22594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f16c3ae8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f16c3af2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f16bda960d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f16bda8b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f16be8ef6db start_thread at ??:0
@ 0x7f16ba6c661f clone at ??:0
W20260927 21:10:38.087668 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:38.087898 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:38.087934 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:38.089057 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:38.089120 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:38.089165 16864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:10:38.090592 16864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36335
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:38.090859 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:38.091128 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:38.095958 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:38.095956 16904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:38.096138 16864 server_base.cc:1034] running on GCE node
W20260927 21:10:38.096076 16906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:38.096477 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:38.096766 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:38.097924 16864 hybrid_clock.cc:648] HybridClock initialized: now 1790543438097913 us; error 27 us; skew 500 ppm
Sep 27 21:10:38 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:38.100567 16864 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:10:38.100979 16864 webserver.cc:460] Webserver started at http://127.0.98.1:36335/ using document root <none> and password file <none>
I20260927 21:10:38.101202 16864 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:38.101276 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:38.102667 16864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:38.103433 16913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.103628 16864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:38.103739 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "4f6bd53f531b4610b0cc0b63e28f23c6"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "e726581ed4c746fcce303c98c08aca01"
server_key_iv: "0311cb76e4f2843c97003ca6fdf899a9"
server_key_version: "encryptionkey@0"
I20260927 21:10:38.104092 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:38.134908 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:38.137272 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:38.137445 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:38.137688 16864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:38.138121 16920 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:10:38.140096 16864 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:10:38.140159 16864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:10:38.140208 16864 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:10:38.140842 16864 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:10:38.140897 16864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:10:38.140977 16920 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap starting.
I20260927 21:10:38.148132 16864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44837
I20260927 21:10:38.148200 17027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44837 every 8 connection(s)
I20260927 21:10:38.148554 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:10:38.152001 16920 log.cc:824] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:38.154098 16920 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:10:38.154367 16920 tablet_bootstrap.cc:492] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Bootstrap complete.
I20260927 21:10:38.155371 16920 ts_tablet_manager.cc:1397] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.009s
I20260927 21:10:38.156019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 16864
I20260927 21:10:38.156307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39159
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46089
--webserver_port=34439
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:10:38.156906 16920 raft_consensus.cc:348] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:38.157279 16920 raft_consensus.cc:729] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6bd53f531b4610b0cc0b63e28f23c6, State: Initialized, Role: FOLLOWER
I20260927 21:10:38.157449 16920 consensus_queue.cc:260] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:38.157727 16920 ts_tablet_manager.cc:1428] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260927 21:10:38.195845 17030 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:10:38.149242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:32985)
0927 21:10:38.149416 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.149459 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:38.149904 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:38.195631 (+ 45727us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
W20260927 21:10:38.196126 17028 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46089 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46089: Client connection negotiation failed: client connection to 127.0.98.62:46089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:10:38.196498 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.196379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:49387)
0927 21:10:38.196431 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.196473 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:10:38.196822 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.196743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:47343)
0927 21:10:38.196784 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.196807 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 21:10:38.196945 17028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:10:38.238044 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:38.238234 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:38.238266 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:38.239418 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:38.239459 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:38.239485 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:10:38.240857 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34439
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:38.241027 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:38.241187 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:38.245997 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:38.245990 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:38.246115 17032 server_base.cc:1034] running on GCE node
W20260927 21:10:38.246202 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:38.246402 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:38.246640 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:38.247800 17032 hybrid_clock.cc:648] HybridClock initialized: now 1790543438247800 us; error 40 us; skew 500 ppm
Sep 27 21:10:38 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:38.250036 17032 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:10:38.250490 17032 webserver.cc:460] Webserver started at http://127.0.98.2:34439/ using document root <none> and password file <none>
I20260927 21:10:38.250718 17032 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:38.250788 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:38.252103 17032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:38.252637 17048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.252790 17032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:38.252882 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "70bceee4b3464d49b25455d0330ca9a1"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "4c7d66afd255e7acaebbef70b66cdbea"
server_key_iv: "ca1a9e444a92db4a8222a44f401b03bd"
server_key_version: "encryptionkey@0"
I20260927 21:10:38.253209 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:38.264624 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:38.266322 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:38.266479 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:38.266697 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:38.266989 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:38.267045 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.267122 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:38.267150 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.273010 17032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39159
I20260927 21:10:38.273047 17161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39159 every 8 connection(s)
I20260927 21:10:38.273378 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:10:38.274377 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.274029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:48929)
0927 21:10:38.274180 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.274248 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260927 21:10:38.274778 17162 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46089 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46089: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
I20260927 21:10:38.275024 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.274943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:56631)
0927 21:10:38.274983 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.275008 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 21:10:38.275327 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.275243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:35935)
0927 21:10:38.275292 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.275314 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 21:10:38.275453 17162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:10:38.281457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17032
I20260927 21:10:38.281716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45733
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46089
--webserver_port=38663
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:38.362635 17165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:38.362826 17165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:38.362848 17165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:38.363934 17165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:38.363969 17165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:38.363994 17165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:10:38.365331 17165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38663
--tserver_master_addrs=127.0.98.62:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:38.365504 17165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:38.365684 17165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:38.370496 17171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:38.370630 17165 server_base.cc:1034] running on GCE node
W20260927 21:10:38.370492 17170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:38.370702 17173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:38.370954 17165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:38.371250 17165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:38.372393 17165 hybrid_clock.cc:648] HybridClock initialized: now 1790543438372383 us; error 27 us; skew 500 ppm
Sep 27 21:10:38 dist-test-slave-8bgs krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:38.374919 17165 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:10:38.375434 17165 webserver.cc:460] Webserver started at http://127.0.98.3:38663/ using document root <none> and password file <none>
I20260927 21:10:38.375684 17165 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:38.375736 17165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:38.377051 17165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:10:38.377640 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.377839 17165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:10:38.377924 17165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "bfabea23458a4306a84b14121a378304"
format_stamp: "Formatted at 2026-09-27 21:09:56 on dist-test-slave-8bgs"
server_key: "f029c4b01788b376c3f18f07c0dd111e"
server_key_iv: "a4338a9959104a7132a07e0960a89bcc"
server_key_version: "encryptionkey@0"
I20260927 21:10:38.378254 17165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:38.390787 17165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:38.393695 17165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:38.393847 17165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:38.394093 17165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:38.394475 17165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:38.394520 17165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.394557 17165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:38.394582 17165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:38.400507 17165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45733
I20260927 21:10:38.400543 17293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45733 every 8 connection(s)
I20260927 21:10:38.400895 17165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:10:38.401906 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.401557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:33913)
0927 21:10:38.401705 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.401770 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260927 21:10:38.402315 17294 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46089 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46089: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
I20260927 21:10:38.402573 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.402487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:37863)
0927 21:10:38.402531 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.402556 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 21:10:38.402881 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:38.402809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:41219)
0927 21:10:38.402848 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:38.402868 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:10:38.402993 17294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:10:38.406574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17165
I20260927 21:10:39.197504 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:39.197293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:36591)
0927 21:10:39.197408 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:39.197441 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260927 21:10:39.275885 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:39.275701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:49807)
0927 21:10:39.275808 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:39.275840 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260927 21:10:39.403498 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:39.403314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:46563)
0927 21:10:39.403424 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:39.403454 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 21:10:40.024271 17301 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:40.024392 17301 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:40.024786 17301 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
Sep 27 21:10:40 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 21:10:40 dist-test-slave-8bgs krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 21:10:40.040359 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.025163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44287 (local address 127.0.98.2:39159)
0927 21:10:40.025286 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:40.025296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:40.025322 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:10:40.025730 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:40.025732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:40.025768 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:40.025871 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:40.026678 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.027663 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.028643 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.029046 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.029162 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:40.036384 (+ 7222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:40.036389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:40.036389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:40.036400 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:40.038656 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:40.039138 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:40.039141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:40.039141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:40.039235 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:40.039495 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:40.039496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:40.039496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:40.039732 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:40.039820 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:40.040106 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 21:10:40.040238 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260927 21:10:40.040374 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.025080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33219 (local address 127.0.98.3:45733)
0927 21:10:40.025258 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:40.025278 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:40.025298 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:10:40.025721 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:40.025724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:40.025764 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:40.025915 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:40.026720 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.027660 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.028640 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.029029 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.029156 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:40.036724 (+ 7568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:40.036728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:40.036728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:40.036742 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:40.038660 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:40.039125 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:40.039128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:40.039128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:40.039229 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:40.039472 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:40.039473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:40.039474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:40.039733 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:40.039823 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:40.040099 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 21:10:40.040251 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":58,"threads_started":1}
I20260927 21:10:40.040375 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.024948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45733 (local address 127.0.98.1:33219)
0927 21:10:40.025101 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:40.025113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:40.025177 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:40.025946 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:40.025948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:40.026340 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:40.026554 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:40.026560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.027762 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.027763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:40.028538 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:40.028540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.029129 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.029130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:40.029247 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:40.034764 (+ 5517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:40.034773 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:40.036616 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:40.038737 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:40.038743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:40.038745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:40.039011 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:40.039298 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:40.039299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:40.039299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:40.039407 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:40.039807 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:40.039813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:40.039931 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 21:10:40.040041 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:10:40.040235 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":5,"thread_start_us":104,"threads_started":1}
I20260927 21:10:40.040732 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.024972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39159 (local address 127.0.98.1:44287)
0927 21:10:40.025109 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:40.025121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:40.025177 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:40.025971 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:40.025971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:40.026342 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:40.026553 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:40.026560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.027769 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.027770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:40.028540 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:40.028542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:40.029136 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:40.029137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:40.029244 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:40.034772 (+ 5528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:40.034777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:40.036247 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:40.038775 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:40.038776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:40.038776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:40.039011 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:40.039324 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:40.039325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:40.039325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:40.039410 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:40.039814 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:40.039815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:40.039931 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 21:10:40.040042 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:10:40.040318 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":183,"thread_start_us":61,"threads_started":1}
W20260927 21:10:40.041189 16917 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
W20260927 21:10:40.041550 16914 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:40.041625 16914 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6; no voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304
I20260927 21:10:40.041746 17301 raft_consensus.cc:2726] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:10:40.198012 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.197875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:38919)
0927 21:10:40.197937 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:40.197971 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:10:40.276618 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.276332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:51743)
0927 21:10:40.276531 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:40.276578 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260927 21:10:40.404126 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:40.403919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:49537)
0927 21:10:40.404030 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:40.404071 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260927 21:10:41.198721 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:41.198467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:34291)
0927 21:10:41.198609 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:41.198648 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260927 21:10:41.277276 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:41.277097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:35787)
0927 21:10:41.277203 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:41.277237 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260927 21:10:41.404717 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:41.404528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:51581)
0927 21:10:41.404650 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:41.404678 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260927 21:10:41.820554 17311 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:41.820636 17311 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:41.820801 17311 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
W20260927 21:10:41.821233 16914 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
W20260927 21:10:41.821319 16917 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:41.821426 16917 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6; no voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304
I20260927 21:10:41.821507 17311 raft_consensus.cc:2726] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:10:42.199458 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:42.199252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:46429)
0927 21:10:42.199381 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:42.199410 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260927 21:10:42.278048 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:42.277844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:52049)
0927 21:10:42.277964 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:42.277999 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260927 21:10:42.405333 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:42.405140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:38005)
0927 21:10:42.405261 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:42.405300 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260927 21:10:43.200132 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:43.199936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:57645)
0927 21:10:43.200052 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:43.200085 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260927 21:10:43.278659 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:43.278470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:34559)
0927 21:10:43.278588 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:43.278622 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 21:10:43.405946 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:43.405771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:48265)
0927 21:10:43.405871 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:43.405907 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
I20260927 21:10:44.200755 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:44.200546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:36425)
0927 21:10:44.200670 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:44.200707 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260927 21:10:44.279335 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:44.279146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:50041)
0927 21:10:44.279260 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:44.279296 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260927 21:10:44.406553 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:44.406340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:34749)
0927 21:10:44.406485 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:44.406518 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260927 21:10:44.884964 17321 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:44.885059 17321 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:44.885334 17321 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
W20260927 21:10:44.885583 16917 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
W20260927 21:10:44.885663 16914 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:44.885761 16914 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6; no voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304
I20260927 21:10:44.885839 17321 raft_consensus.cc:2726] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:10:45.201439 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:45.201233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:38837)
0927 21:10:45.201359 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:45.201394 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260927 21:10:45.279961 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:45.279766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:49869)
0927 21:10:45.279884 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:45.279921 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260927 21:10:45.407287 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:45.407005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:42919)
0927 21:10:45.407168 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:45.407239 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":147,"threads_started":1}
I20260927 21:10:46.202113 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:46.201871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:44955)
0927 21:10:46.202036 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:46.202067 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260927 21:10:46.280535 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:46.280348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:59273)
0927 21:10:46.280468 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:46.280499 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260927 21:10:46.407986 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:46.407741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:58795)
0927 21:10:46.407908 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:46.407946 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":58,"threads_started":1}
I20260927 21:10:47.202832 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:47.202582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:43833)
0927 21:10:47.202749 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:47.202787 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260927 21:10:47.274721 17329 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:47.274808 17329 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:47.275003 17329 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
W20260927 21:10:47.275442 16917 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
W20260927 21:10:47.275523 16914 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:47.275553 16914 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6; no voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304
I20260927 21:10:47.275633 17329 raft_consensus.cc:2726] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:10:47.281141 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:47.280906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:51115)
0927 21:10:47.281024 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:47.281056 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260927 21:10:47.408588 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:47.408404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:59127)
0927 21:10:47.408513 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:47.408549 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260927 21:10:48.203735 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:48.203470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:44957)
0927 21:10:48.203640 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:48.203687 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260927 21:10:48.281821 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:48.281554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:35427)
0927 21:10:48.281726 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:48.281777 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260927 21:10:48.409257 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:48.409056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:49151)
0927 21:10:48.409172 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:48.409215 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260927 21:10:49.113312 17335 raft_consensus.cc:482] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:49.113412 17335 raft_consensus.cc:504] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70bceee4b3464d49b25455d0330ca9a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39159 } } peers { permanent_uuid: "bfabea23458a4306a84b14121a378304" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45733 } } peers { permanent_uuid: "4f6bd53f531b4610b0cc0b63e28f23c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44837 } }
I20260927 21:10:49.113651 17335 leader_election.cc:288] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159), bfabea23458a4306a84b14121a378304 (127.0.98.3:45733)
W20260927 21:10:49.114044 16914 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70bceee4b3464d49b25455d0330ca9a1 (127.0.98.2:39159): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
W20260927 21:10:49.114125 16917 leader_election.cc:341] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfabea23458a4306a84b14121a378304 (127.0.98.3:45733): Not found: Tablet not found: 973c54646ce44a5f8cba0d62f91408b7
I20260927 21:10:49.114176 16917 leader_election.cc:302] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f6bd53f531b4610b0cc0b63e28f23c6; no voters: 70bceee4b3464d49b25455d0330ca9a1, bfabea23458a4306a84b14121a378304
I20260927 21:10:49.114244 17335 raft_consensus.cc:2726] T 973c54646ce44a5f8cba0d62f91408b7 P 4f6bd53f531b4610b0cc0b63e28f23c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:10:49.204653 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:49.204379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:55143)
0927 21:10:49.204548 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:49.204594 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260927 21:10:49.282526 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:49.282311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:54631)
0927 21:10:49.282435 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:49.282483 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260927 21:10:49.409919 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:49.409696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:39567)
0927 21:10:49.409829 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:49.409878 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260927 21:10:50.205475 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:50.205217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:34869)
0927 21:10:50.205373 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:50.205427 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260927 21:10:50.283140 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:50.282915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:58953)
0927 21:10:50.283023 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:50.283055 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260927 21:10:50.410576 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:50.410387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:53295)
0927 21:10:50.410505 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:50.410536 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260927 21:10:51.206156 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:51.205950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:55423)
0927 21:10:51.206076 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:51.206109 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260927 21:10:51.283779 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:51.283587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:47983)
0927 21:10:51.283700 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:51.283736 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260927 21:10:51.411243 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:51.411010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:58579)
0927 21:10:51.411149 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:51.411199 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260927 21:10:52.206854 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:52.206609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:58589)
0927 21:10:52.206775 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:52.206806 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260927 21:10:52.284490 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:52.284233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:51397)
0927 21:10:52.284389 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:52.284443 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260927 21:10:52.411846 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:52.411658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:59661)
0927 21:10:52.411772 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:52.411806 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260927 21:10:53.207549 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:53.207350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.1:41441)
0927 21:10:53.207477 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:53.207504 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260927 21:10:53.285187 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:53.284982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.2:33819)
0927 21:10:53.285107 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:53.285143 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260927 21:10:53.412532 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:53.412333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46089 (local address 127.0.98.3:54781)
0927 21:10:53.412459 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:53.412493 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260927 21:10:53.415926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 16864
I20260927 21:10:53.422676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17032
I20260927 21:10:53.427312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17165
2026-09-27T21:10:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58816 ms)
[----------] 30 tests from SecurityITest (232489 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:10:53 dist-test-slave-8bgs krb5kdc[17353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:10:53 dist-test-slave-8bgs krb5kdc[17353](info): set up 2 sockets
Sep 27 21:10:53 dist-test-slave-8bgs krb5kdc[17353](info): commencing operation
krb5kdc: starting...
W20260927 21:10:54.454707 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:10:54 dist-test-slave-8bgs krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:10:54.566152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:54.645203 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:54.645427 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:54.645465 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:54.646560 17366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:10:54.646633 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:54.646664 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:54.646693 17366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:10:54.646723 17366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:10:54.648273 17366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:54.648541 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:54.648753 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:54.653541 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:54.653527 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:54.653517 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:54.653587 17366 server_base.cc:1034] running on GCE node
I20260927 21:10:54.654009 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:54.654330 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:54.655465 17366 hybrid_clock.cc:648] HybridClock initialized: now 1790543454655443 us; error 39 us; skew 500 ppm
Sep 27 21:10:54 dist-test-slave-8bgs krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:54.663823 17366 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:10:54.664333 17366 webserver.cc:460] Webserver started at http://127.0.98.62:46251/ using document root <none> and password file <none>
I20260927 21:10:54.664567 17366 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:54.664635 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:54.664800 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:54.665918 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "ebc460da17b34498aab67b212d2b52ad"
format_stamp: "Formatted at 2026-09-27 21:10:54 on dist-test-slave-8bgs"
server_key: "14ff30acbff2d0d79fc79189642da48b"
server_key_iv: "49a2041927ea1fb0a98dfc97cd152058"
server_key_version: "encryptionkey@0"
I20260927 21:10:54.666297 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebc460da17b34498aab67b212d2b52ad"
format_stamp: "Formatted at 2026-09-27 21:10:54 on dist-test-slave-8bgs"
server_key: "14ff30acbff2d0d79fc79189642da48b"
server_key_iv: "49a2041927ea1fb0a98dfc97cd152058"
server_key_version: "encryptionkey@0"
I20260927 21:10:54.667748 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260927 21:10:54.668551 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:54.668767 17366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:54.668874 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "ebc460da17b34498aab67b212d2b52ad"
format_stamp: "Formatted at 2026-09-27 21:10:54 on dist-test-slave-8bgs"
server_key: "14ff30acbff2d0d79fc79189642da48b"
server_key_iv: "49a2041927ea1fb0a98dfc97cd152058"
server_key_version: "encryptionkey@0"
I20260927 21:10:54.668953 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:54.684331 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:54.686129 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:54.686302 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:54.690402 17366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37229
I20260927 21:10:54.690454 17433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37229 every 8 connection(s)
I20260927 21:10:54.690819 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:10:54.691672 17434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:54.693892 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad: Bootstrap starting.
I20260927 21:10:54.694460 17434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:54.694747 17434 log.cc:824] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:54.695446 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad: No bootstrap required, opened a new log
I20260927 21:10:54.697016 17434 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc460da17b34498aab67b212d2b52ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37229 } }
I20260927 21:10:54.697161 17434 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:54.697185 17434 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc460da17b34498aab67b212d2b52ad, State: Initialized, Role: FOLLOWER
I20260927 21:10:54.697345 17434 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc460da17b34498aab67b212d2b52ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37229 } }
I20260927 21:10:54.697407 17434 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:10:54.697448 17434 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:10:54.697495 17434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:54.698001 17434 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc460da17b34498aab67b212d2b52ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37229 } }
I20260927 21:10:54.698123 17434 leader_election.cc:302] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc460da17b34498aab67b212d2b52ad; no voters:
I20260927 21:10:54.698298 17434 leader_election.cc:288] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:10:54.698344 17437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:54.698518 17437 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [term 1 LEADER]: Becoming Leader. State: Replica: ebc460da17b34498aab67b212d2b52ad, State: Running, Role: LEADER
I20260927 21:10:54.698556 17434 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:10:54.698660 17437 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc460da17b34498aab67b212d2b52ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37229 } }
I20260927 21:10:54.699054 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc460da17b34498aab67b212d2b52ad. Latest consensus state: current_term: 1 leader_uuid: "ebc460da17b34498aab67b212d2b52ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc460da17b34498aab67b212d2b52ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37229 } } }
I20260927 21:10:54.699190 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:54.699196 17438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc460da17b34498aab67b212d2b52ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc460da17b34498aab67b212d2b52ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37229 } } }
I20260927 21:10:54.699342 17438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:54.699915 17446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:10:54.700512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17366
I20260927 21:10:54.700621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:10:54.700664 17446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:10:54.700834 392 external_mini_cluster.cc:1428] Setting key 3ed51a8695d8fafdb5edbba34e078ea1
I20260927 21:10:54.702211 17446 catalog_manager.cc:1347] Generated new cluster ID: eefbf55c593c413ba737f731f27058ce
I20260927 21:10:54.702265 17446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:10:54 dist-test-slave-8bgs krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:54.709323 17446 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:10:54.710023 17446 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:10:54.712579 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:54.701303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.0.98.62:37229)
0927 21:10:54.701481 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:54.701491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:54.701687 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:10:54.701810 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:54.701817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:54.701879 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:54.702021 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:54.702825 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:54.704199 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:54.705308 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:54.705884 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:54.705980 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:54.708904 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:54.708908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:54.708908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:54.708917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:54.710962 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:54.711405 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:54.711408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:54.711408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:54.711509 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:54.711713 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:54.711714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:54.711714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:54.711962 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:54.712120 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:54.712220 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:10:54.712360 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260927 21:10:54.716418 17446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad: Generated new TSK 0
I20260927 21:10:54.716600 17446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:10:54.749545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37229
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:54.844379 17460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:54.844589 17460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:54.844614 17460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:54.845808 17460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:54.845849 17460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:54.845875 17460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:10:54.847419 17460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:54.847653 17460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:54.847892 17460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:54.853461 17466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:54.853466 17465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:54.853664 17468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:54.853827 17460 server_base.cc:1034] running on GCE node
I20260927 21:10:54.854027 17460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:54.854321 17460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:54.855477 17460 hybrid_clock.cc:648] HybridClock initialized: now 1790543454855463 us; error 26 us; skew 500 ppm
Sep 27 21:10:54 dist-test-slave-8bgs krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:54.857951 17460 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:10:54.858388 17460 webserver.cc:460] Webserver started at http://127.0.98.1:36145/ using document root <none> and password file <none>
I20260927 21:10:54.858565 17460 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:54.858605 17460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:54.858688 17460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:54.859745 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "d71ad10b8fd4420ea6fde70033eb044f"
format_stamp: "Formatted at 2026-09-27 21:10:54 on dist-test-slave-8bgs"
server_key: "2411747721563d9506d68234fd7d4fa8"
server_key_iv: "041c19a09e1ffe1b6bc8abe643878924"
server_key_version: "encryptionkey@0"
I20260927 21:10:54.860054 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d71ad10b8fd4420ea6fde70033eb044f"
format_stamp: "Formatted at 2026-09-27 21:10:54 on dist-test-slave-8bgs"
server_key: "2411747721563d9506d68234fd7d4fa8"
server_key_iv: "041c19a09e1ffe1b6bc8abe643878924"
server_key_version: "encryptionkey@0"
I20260927 21:10:54.861282 17460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:10:54.861949 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:54.862119 17460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:54.862223 17460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "d71ad10b8fd4420ea6fde70033eb044f"
format_stamp: "Formatted at 2026-09-27 21:10:54 on dist-test-slave-8bgs"
server_key: "2411747721563d9506d68234fd7d4fa8"
server_key_iv: "041c19a09e1ffe1b6bc8abe643878924"
server_key_version: "encryptionkey@0"
I20260927 21:10:54.862295 17460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:54.899000 17460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:54.901242 17460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:54.901408 17460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:54.901657 17460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:54.901988 17460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:54.902043 17460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:54.902084 17460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:54.902110 17460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:54.907979 17460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39021
I20260927 21:10:54.908032 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39021 every 8 connection(s)
I20260927 21:10:54.908347 17460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:10:54 dist-test-slave-8bgs krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:54.915452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17460
I20260927 21:10:54.915581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:10:54.915846 392 external_mini_cluster.cc:1428] Setting key 0e3b5e5d0b7c17bf2cfca81ed7576582
I20260927 21:10:54.918175 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:54.909025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33193 (local address 127.0.98.62:37229)
0927 21:10:54.909087 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:54.909089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:54.909326 (+ 237us) server_negotiation.cc:407] Connection header received
0927 21:10:54.910278 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:54.910280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:54.910292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:54.910341 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:54.910984 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:54.911556 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:54.912585 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:54.912792 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:54.912874 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:54.914999 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:54.915000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:54.915001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:54.915008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:54.916600 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:54.917088 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:54.917089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:54.917089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:54.917142 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:54.917432 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:54.917432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:54.917432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:54.917535 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:54.917587 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:54.917939 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 21:10:54.918049 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:10:54.918290 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:54.909004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37229 (local address 127.0.98.1:33193)
0927 21:10:54.909192 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:54.909231 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:54.910168 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:54.910412 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:54.910416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:54.910678 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:54.910883 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:54.910887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:54.911654 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:54.911655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:54.912485 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:54.912488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:54.912881 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:54.912883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:54.913012 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:54.913486 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:54.913494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:54.914906 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:54.916686 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:54.916694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:54.916697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:54.917011 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:54.917210 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:54.917211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:54.917212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:54.917366 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:54.917599 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:54.917605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:54.917772 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:10:54.917888 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:10:54.918090 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260927 21:10:54.918802 17589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37229
I20260927 21:10:54.918912 17589 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:54.919155 17589 heartbeater.cc:511] Master 127.0.98.62:37229 requested a full tablet report, sending...
I20260927 21:10:54.919744 17398 ts_manager.cc:194] Registered new tserver with Master: d71ad10b8fd4420ea6fde70033eb044f (127.0.98.1:39021)
I20260927 21:10:54.920426 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33193
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:10:54.941497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37229
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:55.021898 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:55.022151 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:55.022190 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:55.023417 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:55.023494 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:55.023537 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:10:55.025089 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:55.025370 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:55.025609 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:55.030610 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.030610 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.030683 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.031139 17594 server_base.cc:1034] running on GCE node
I20260927 21:10:55.031356 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:55.031559 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:55.032701 17594 hybrid_clock.cc:648] HybridClock initialized: now 1790543455032689 us; error 26 us; skew 500 ppm
Sep 27 21:10:55 dist-test-slave-8bgs krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:55.035607 17594 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:10:55.036118 17594 webserver.cc:460] Webserver started at http://127.0.98.2:38913/ using document root <none> and password file <none>
I20260927 21:10:55.036355 17594 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:55.036401 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:55.036588 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:55.037670 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ebaf898a745408d9e1511fc8f445041"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "edb91bb6fc7b272708df39920672976d"
server_key_iv: "563606705d868108041eefc1e2cba987"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.038038 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ebaf898a745408d9e1511fc8f445041"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "edb91bb6fc7b272708df39920672976d"
server_key_iv: "563606705d868108041eefc1e2cba987"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.039397 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:10:55.040138 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.040321 17594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:10:55.040413 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "0ebaf898a745408d9e1511fc8f445041"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "edb91bb6fc7b272708df39920672976d"
server_key_iv: "563606705d868108041eefc1e2cba987"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.040494 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:55.049847 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:55.051921 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:55.052075 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:55.052294 17594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:55.052598 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:55.052656 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.052697 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:55.052733 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.058584 17594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34847
I20260927 21:10:55.058627 17722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34847 every 8 connection(s)
I20260927 21:10:55.058955 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:10:55 dist-test-slave-8bgs krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:55.066025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17594
I20260927 21:10:55.066143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:10:55.066363 392 external_mini_cluster.cc:1428] Setting key c793319cd6510d0d22f513b82c58bd47
I20260927 21:10:55.068439 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.059603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40497 (local address 127.0.98.62:37229)
0927 21:10:55.059673 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.059675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.059897 (+ 222us) server_negotiation.cc:407] Connection header received
0927 21:10:55.060836 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.060838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.060852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.060908 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.061541 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.062051 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.063024 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.063274 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.063330 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.065375 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.065377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.065377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:55.065385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.066964 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:55.067522 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:55.067523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:55.067523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:55.067575 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:55.067855 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:55.067856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:55.067856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:55.067953 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.068003 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.068308 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.068377 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:10:55.068738 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.059587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37229 (local address 127.0.98.2:40497)
0927 21:10:55.059767 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.059807 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.060737 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.060989 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.060992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.061251 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:55.061442 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.061446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.062133 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.062134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.062942 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.062944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.063344 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.063345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.063443 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.063857 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:55.063865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:55.065274 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:55.067073 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:55.067078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:55.067123 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:55.067442 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:55.067640 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:55.067641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:55.067642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:55.067793 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:55.068014 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:55.068019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:55.068181 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.068291 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.068460 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":115,"threads_started":1}
I20260927 21:10:55.069291 17723 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37229
I20260927 21:10:55.069396 17723 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:55.069584 17723 heartbeater.cc:511] Master 127.0.98.62:37229 requested a full tablet report, sending...
I20260927 21:10:55.070006 17398 ts_manager.cc:194] Registered new tserver with Master: 0ebaf898a745408d9e1511fc8f445041 (127.0.98.2:34847)
I20260927 21:10:55.070530 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40497
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:10:55.091758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37229
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:55.171801 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:55.172046 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:55.172086 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:55.173273 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:55.173321 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:55.173348 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:10:55.174762 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:55.174948 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:55.175202 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:55.180074 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.180086 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.180049 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.180356 17728 server_base.cc:1034] running on GCE node
I20260927 21:10:55.180558 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:55.180743 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:55.181879 17728 hybrid_clock.cc:648] HybridClock initialized: now 1790543455181868 us; error 25 us; skew 500 ppm
Sep 27 21:10:55 dist-test-slave-8bgs krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:55.184492 17728 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:10:55.184988 17728 webserver.cc:460] Webserver started at http://127.0.98.3:45619/ using document root <none> and password file <none>
I20260927 21:10:55.185210 17728 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:55.185278 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:55.185438 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:55.186474 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "05e77ca243a24ca8b3c56518c0470458"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "1566c92bdf36b4b337fddd18846a3a7c"
server_key_iv: "e9410d7bee6532eb0bda1ff0b3423ed8"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.186832 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "05e77ca243a24ca8b3c56518c0470458"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "1566c92bdf36b4b337fddd18846a3a7c"
server_key_iv: "e9410d7bee6532eb0bda1ff0b3423ed8"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.188268 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:10:55.188997 17743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.189170 17728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:55.189270 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "05e77ca243a24ca8b3c56518c0470458"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "1566c92bdf36b4b337fddd18846a3a7c"
server_key_iv: "e9410d7bee6532eb0bda1ff0b3423ed8"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.189364 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:55.205366 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:55.207729 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:55.207901 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:55.208140 17728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:55.208456 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:55.208511 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.208554 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:55.208581 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.214720 17728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38129
I20260927 21:10:55.214818 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38129 every 8 connection(s)
I20260927 21:10:55.215160 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:10:55.216214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17728
I20260927 21:10:55.216344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:10:55.216523 392 external_mini_cluster.cc:1428] Setting key 3f4ce301f51c9e991dd7f732ae401056
Sep 27 21:10:55 dist-test-slave-8bgs krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:55.224380 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.215876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57741 (local address 127.0.98.62:37229)
0927 21:10:55.215931 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.215933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.216224 (+ 291us) server_negotiation.cc:407] Connection header received
0927 21:10:55.217515 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.217516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.217525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.217562 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.218141 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.218624 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.219725 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.219923 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.219978 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.221927 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.221928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.221929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:55.221934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.223217 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:55.223575 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:55.223576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:55.223576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:55.223625 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:55.223846 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:55.223847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:55.223847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:55.223947 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.223996 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.224256 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.224314 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:10:55.224567 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.215859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37229 (local address 127.0.98.3:57741)
0927 21:10:55.216053 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.216104 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.217432 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.217618 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.217621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.217877 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:55.218067 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.218071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.218738 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.218739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.219648 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.219651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.219999 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.220000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.220097 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.220522 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:55.220531 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:55.221842 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:55.223284 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:55.223290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:55.223292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:55.223514 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:55.223680 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:55.223680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:55.223681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:55.223773 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:55.224027 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:55.224031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:55.224146 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.224229 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.224344 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260927 21:10:55.225169 17857 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37229
I20260927 21:10:55.225284 17857 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:55.225466 17857 heartbeater.cc:511] Master 127.0.98.62:37229 requested a full tablet report, sending...
I20260927 21:10:55.225868 17398 ts_manager.cc:194] Registered new tserver with Master: 05e77ca243a24ca8b3c56518c0470458 (127.0.98.3:38129)
I20260927 21:10:55.226225 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57741
I20260927 21:10:55.228127 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:10:55.234943 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.229356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.98.62:37229)
0927 21:10:55.229410 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.229412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.229544 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:10:55.229581 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.229582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.229590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.229634 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.230155 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.230607 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.231372 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.231544 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.231596 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.232669 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.232670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.232671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:55.232676 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.233948 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:55.234258 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:55.234259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:55.234259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:55.234308 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:55.234470 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:55.234470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:55.234470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:55.234598 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.234670 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.234762 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.234817 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:10:55.236549 17398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:10:55.238143 17398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:10:55.244809 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.240435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38129 (local address 127.0.0.1:60130)
0927 21:10:55.240743 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.240754 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.240809 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.241125 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.241129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.241132 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.241208 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.241211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.242607 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.242608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.243745 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.243747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.244486 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.244487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.244565 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.244579 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.244645 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.244717 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":85,"threads_started":1}
I20260927 21:10:55.244879 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.240556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.0.98.2:34847)
0927 21:10:55.240926 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.240930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.240938 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:10:55.241172 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.241175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.241197 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.241455 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.241542 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.242230 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.243794 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.244522 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.244600 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.244659 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.244738 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.244778 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":73,"threads_started":1}
I20260927 21:10:55.245044 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.240672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.3:38129)
0927 21:10:55.240859 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.240864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.240875 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:10:55.240918 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.240920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.240938 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.241017 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.241290 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.242500 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.243848 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.244389 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.244484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.244546 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.244906 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.244960 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260927 21:10:55.245204 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.240870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.0.98.1:39021)
0927 21:10:55.241073 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.241076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.241229 (+ 153us) server_negotiation.cc:407] Connection header received
0927 21:10:55.241272 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.241275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.241290 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.241375 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.241610 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.242673 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.243849 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.244577 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.244633 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.244679 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.245045 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.245114 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260927 21:10:55.244819 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.240424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34847 (local address 127.0.0.1:54714)
0927 21:10:55.240541 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.240582 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.240642 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.241325 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.241326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.241328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.241380 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.241382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.242340 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.242341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.243687 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.243690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.244597 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.244597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.244660 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.244663 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.244727 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.244761 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":31,"thread_start_us":72,"threads_started":1}
I20260927 21:10:55.244984 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.240645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39021 (local address 127.0.0.1:37552)
0927 21:10:55.241049 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.241059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.241171 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.241468 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.241468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.241470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.241521 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.241522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.242767 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.242768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.243721 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.243723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.244728 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.244729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.244796 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.244801 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.244861 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.244905 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":70,"threads_started":1}
I20260927 21:10:55.245954 17791 tablet_service.cc:1467] Processing CreateTablet for tablet 8c234572742e4c76b682c433a2bf11ee (DEFAULT_TABLE table=test-table [id=f42b1926886a411ab77d0ee4e0d82baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:55.246008 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 8c234572742e4c76b682c433a2bf11ee (DEFAULT_TABLE table=test-table [id=f42b1926886a411ab77d0ee4e0d82baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:55.246290 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c234572742e4c76b682c433a2bf11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.246312 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c234572742e4c76b682c433a2bf11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.247684 17657 tablet_service.cc:1467] Processing CreateTablet for tablet 8c234572742e4c76b682c433a2bf11ee (DEFAULT_TABLE table=test-table [id=f42b1926886a411ab77d0ee4e0d82baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:55.248014 17657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c234572742e4c76b682c433a2bf11ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.249202 17874 tablet_bootstrap.cc:492] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Bootstrap starting.
I20260927 21:10:55.249871 17874 tablet_bootstrap.cc:654] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.250458 17874 log.cc:824] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.250769 17876 tablet_bootstrap.cc:492] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Bootstrap starting.
I20260927 21:10:55.250918 17875 tablet_bootstrap.cc:492] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f: Bootstrap starting.
I20260927 21:10:55.251395 17874 tablet_bootstrap.cc:492] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: No bootstrap required, opened a new log
I20260927 21:10:55.251474 17874 ts_tablet_manager.cc:1397] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:10:55.251493 17876 tablet_bootstrap.cc:654] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.251559 17875 tablet_bootstrap.cc:654] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.252120 17876 log.cc:824] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.252221 17875 log.cc:824] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.252823 17876 tablet_bootstrap.cc:492] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: No bootstrap required, opened a new log
I20260927 21:10:55.252884 17876 ts_tablet_manager.cc:1397] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:10:55.253294 17874 raft_consensus.cc:348] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.253448 17874 raft_consensus.cc:374] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.253492 17874 raft_consensus.cc:729] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e77ca243a24ca8b3c56518c0470458, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.253607 17874 consensus_queue.cc:260] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.253872 17874 ts_tablet_manager.cc:1428] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:10:55.253945 17857 heartbeater.cc:503] Master 127.0.98.62:37229 was elected leader, sending a full tablet report...
I20260927 21:10:55.254534 17876 raft_consensus.cc:348] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.254698 17876 raft_consensus.cc:374] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.254735 17876 raft_consensus.cc:729] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebaf898a745408d9e1511fc8f445041, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.254849 17876 consensus_queue.cc:260] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.255162 17876 ts_tablet_manager.cc:1428] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:10:55.255219 17723 heartbeater.cc:503] Master 127.0.98.62:37229 was elected leader, sending a full tablet report...
I20260927 21:10:55.255440 17875 tablet_bootstrap.cc:492] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f: No bootstrap required, opened a new log
I20260927 21:10:55.255527 17875 ts_tablet_manager.cc:1397] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:10:55.257087 17875 raft_consensus.cc:348] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.257216 17875 raft_consensus.cc:374] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.257241 17875 raft_consensus.cc:729] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71ad10b8fd4420ea6fde70033eb044f, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.257315 17875 consensus_queue.cc:260] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.257496 17875 ts_tablet_manager.cc:1428] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:10:55.257548 17589 heartbeater.cc:503] Master 127.0.98.62:37229 was elected leader, sending a full tablet report...
I20260927 21:10:55.298945 17882 raft_consensus.cc:482] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:55.299145 17882 raft_consensus.cc:504] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.299618 17882 leader_election.cc:288] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ebaf898a745408d9e1511fc8f445041 (127.0.98.2:34847), 05e77ca243a24ca8b3c56518c0470458 (127.0.98.3:38129)
I20260927 21:10:55.303527 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.299898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37647 (local address 127.0.98.2:34847)
0927 21:10:55.299946 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.299948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.300051 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:10:55.300081 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.300082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.300095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.300152 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.300307 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.301055 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.302573 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.303149 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.303224 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.303233 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.303415 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.303454 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:10:55.303527 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.299908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38915 (local address 127.0.98.3:38129)
0927 21:10:55.299954 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.299956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.299965 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:10:55.300041 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.300042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.300054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.300136 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.300307 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.301063 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.302572 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.303150 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.303224 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.303233 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.303415 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.303463 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:10:55.303581 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.299769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38129 (local address 127.0.98.1:38915)
0927 21:10:55.299811 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.299823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.299867 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.300149 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.300150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.300151 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.300225 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.300227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.301143 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.301144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.302475 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.302479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.303238 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.303238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.303337 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.303339 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.303392 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.303449 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":99}
I20260927 21:10:55.303587 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.299783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34847 (local address 127.0.98.1:37647)
0927 21:10:55.299964 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.299972 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.300022 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.300153 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.300153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.300155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.300225 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.300227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.301143 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.301144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.302475 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.302479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.303235 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.303236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.303326 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.303331 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.303392 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.303449 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":80,"thread_start_us":112,"threads_started":1}
I20260927 21:10:55.304020 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c234572742e4c76b682c433a2bf11ee" candidate_uuid: "d71ad10b8fd4420ea6fde70033eb044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebaf898a745408d9e1511fc8f445041" is_pre_election: true
I20260927 21:10:55.304041 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c234572742e4c76b682c433a2bf11ee" candidate_uuid: "d71ad10b8fd4420ea6fde70033eb044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e77ca243a24ca8b3c56518c0470458" is_pre_election: true
I20260927 21:10:55.304145 17677 raft_consensus.cc:2454] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d71ad10b8fd4420ea6fde70033eb044f in term 0.
I20260927 21:10:55.304198 17811 raft_consensus.cc:2454] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d71ad10b8fd4420ea6fde70033eb044f in term 0.
I20260927 21:10:55.304448 17476 leader_election.cc:302] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebaf898a745408d9e1511fc8f445041, d71ad10b8fd4420ea6fde70033eb044f; no voters:
I20260927 21:10:55.304592 17882 raft_consensus.cc:2781] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:10:55.304646 17882 raft_consensus.cc:482] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:10:55.304669 17882 raft_consensus.cc:3037] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:55.305364 17882 raft_consensus.cc:504] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.305516 17882 leader_election.cc:288] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [CANDIDATE]: Term 1 election: Requested vote from peers 0ebaf898a745408d9e1511fc8f445041 (127.0.98.2:34847), 05e77ca243a24ca8b3c56518c0470458 (127.0.98.3:38129)
I20260927 21:10:55.305667 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c234572742e4c76b682c433a2bf11ee" candidate_uuid: "d71ad10b8fd4420ea6fde70033eb044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebaf898a745408d9e1511fc8f445041"
I20260927 21:10:55.305687 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c234572742e4c76b682c433a2bf11ee" candidate_uuid: "d71ad10b8fd4420ea6fde70033eb044f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e77ca243a24ca8b3c56518c0470458"
I20260927 21:10:55.305758 17811 raft_consensus.cc:3037] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:55.305758 17677 raft_consensus.cc:3037] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:55.306629 17811 raft_consensus.cc:2454] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d71ad10b8fd4420ea6fde70033eb044f in term 1.
I20260927 21:10:55.306629 17677 raft_consensus.cc:2454] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d71ad10b8fd4420ea6fde70033eb044f in term 1.
I20260927 21:10:55.306810 17476 leader_election.cc:302] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebaf898a745408d9e1511fc8f445041, d71ad10b8fd4420ea6fde70033eb044f; no voters:
I20260927 21:10:55.306928 17882 raft_consensus.cc:2781] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:55.307130 17882 raft_consensus.cc:686] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [term 1 LEADER]: Becoming Leader. State: Replica: d71ad10b8fd4420ea6fde70033eb044f, State: Running, Role: LEADER
I20260927 21:10:55.307268 17882 consensus_queue.cc:237] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } }
I20260927 21:10:55.308192 17397 catalog_manager.cc:5495] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f reported cstate change: term changed from 0 to 1, leader changed from <none> to d71ad10b8fd4420ea6fde70033eb044f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d71ad10b8fd4420ea6fde70033eb044f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d71ad10b8fd4420ea6fde70033eb044f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39021 } health_report { overall_health: HEALTHY } } }
W20260927 21:10:55.309478 17724 tablet.cc:2369] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:10:55.316722 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.314604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.1:39021)
0927 21:10:55.314650 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.314652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.314659 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:10:55.314721 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.314722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.314737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.314792 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:10:55.314878 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.315396 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.316101 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.316280 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.316332 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.316455 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:10:55.316546 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:10:55.316604 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.316640 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.316655 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:10:55.318365 17677 raft_consensus.cc:1261] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 1 FOLLOWER]: Refusing update from remote peer d71ad10b8fd4420ea6fde70033eb044f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:55.318367 17811 raft_consensus.cc:1261] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 1 FOLLOWER]: Refusing update from remote peer d71ad10b8fd4420ea6fde70033eb044f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:55.318730 17884 consensus_queue.cc:1035] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e77ca243a24ca8b3c56518c0470458" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:55.318895 17882 consensus_queue.cc:1035] T 8c234572742e4c76b682c433a2bf11ee P d71ad10b8fd4420ea6fde70033eb044f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebaf898a745408d9e1511fc8f445041" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:55.320963 17887 mvcc.cc:205] Tried to move back new op lower bound from 7334065992981188608 to 7334065992939122688. Current Snapshot: MvccSnapshot[applied={T|T < 7334065992981188608}]
I20260927 21:10:55.321084 17889 mvcc.cc:205] Tried to move back new op lower bound from 7334065992981188608 to 7334065992939122688. Current Snapshot: MvccSnapshot[applied={T|T < 7334065992981188608}]
I20260927 21:10:55.323266 17398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:55.323354 17398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:55.324085 17398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebc460da17b34498aab67b212d2b52ad: Sending DeleteTablet for 3 replicas of tablet 8c234572742e4c76b682c433a2bf11ee
I20260927 21:10:55.324429 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c234572742e4c76b682c433a2bf11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60130
I20260927 21:10:55.324468 17657 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c234572742e4c76b682c433a2bf11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54714
I20260927 21:10:55.324625 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c234572742e4c76b682c433a2bf11ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37552
I20260927 21:10:55.324646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17460
I20260927 21:10:55.325312 17897 tablet_replica.cc:330] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: stopping tablet replica
I20260927 21:10:55.325418 17897 raft_consensus.cc:2229] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:55.325522 17897 raft_consensus.cc:2258] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:55.325683 17898 tablet_replica.cc:330] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: stopping tablet replica
I20260927 21:10:55.325824 17898 raft_consensus.cc:2229] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:55.325942 17898 raft_consensus.cc:2258] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:55.325968 17897 ts_tablet_manager.cc:1905] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:55.326323 17898 ts_tablet_manager.cc:1905] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:55.327576 17897 ts_tablet_manager.cc:1918] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:55.327678 17897 log.cc:1196] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/8c234572742e4c76b682c433a2bf11ee
I20260927 21:10:55.328012 17897 ts_tablet_manager.cc:1939] T 8c234572742e4c76b682c433a2bf11ee P 05e77ca243a24ca8b3c56518c0470458: Deleting consensus metadata
I20260927 21:10:55.328248 17898 ts_tablet_manager.cc:1918] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:55.328343 17898 log.cc:1196] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/8c234572742e4c76b682c433a2bf11ee
I20260927 21:10:55.328392 17385 catalog_manager.cc:4841] TS 05e77ca243a24ca8b3c56518c0470458 (127.0.98.3:38129): tablet 8c234572742e4c76b682c433a2bf11ee (table test-table [id=f42b1926886a411ab77d0ee4e0d82baa]) successfully deleted
I20260927 21:10:55.328617 17898 ts_tablet_manager.cc:1939] T 8c234572742e4c76b682c433a2bf11ee P 0ebaf898a745408d9e1511fc8f445041: Deleting consensus metadata
I20260927 21:10:55.330049 17382 catalog_manager.cc:4841] TS 0ebaf898a745408d9e1511fc8f445041 (127.0.98.2:34847): tablet 8c234572742e4c76b682c433a2bf11ee (table test-table [id=f42b1926886a411ab77d0ee4e0d82baa]) successfully deleted
W20260927 21:10:55.332953 17382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39021 (error 108)
I20260927 21:10:55.333225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17594
I20260927 21:10:55.333847 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.333320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39021 (local address 127.0.0.1:37572)
0927 21:10:55.333762 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.333798 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
W20260927 21:10:55.335693 17382 catalog_manager.cc:4584] TS d71ad10b8fd4420ea6fde70033eb044f (127.0.98.1:39021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c234572742e4c76b682c433a2bf11ee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39021: connect: Connection refused (error 111)
I20260927 21:10:55.338670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17728
I20260927 21:10:55.342972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17366
2026-09-27T21:10:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1914 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T21:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:10:55Z Disabled control of system clock
I20260927 21:10:55.370167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:10:55.450877 17903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:55.451114 17903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:55.451136 17903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:55.452423 17903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:10:55.452466 17903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:55.452479 17903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:55.452492 17903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:10:55.452545 17903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:10:55.453894 17903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:55.454145 17903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:55.454442 17903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:55.459028 17908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.459136 17911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.459225 17903 server_base.cc:1034] running on GCE node
W20260927 21:10:55.459261 17909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.459515 17903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:55.459800 17903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:55.460964 17903 hybrid_clock.cc:648] HybridClock initialized: now 1790543455460949 us; error 27 us; skew 500 ppm
I20260927 21:10:55.462288 17903 webserver.cc:460] Webserver started at http://127.0.98.62:38171/ using document root <none> and password file <none>
I20260927 21:10:55.462527 17903 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:55.462594 17903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:55.462741 17903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:55.463889 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "6afbf5675df24dc08d71951a021d07b5"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "9d1e13a6f4beb7d4b7b6a22e9e208311"
server_key_iv: "68cdd66c37c2febf51d0b91b89f30cf8"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.464244 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "6afbf5675df24dc08d71951a021d07b5"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "9d1e13a6f4beb7d4b7b6a22e9e208311"
server_key_iv: "68cdd66c37c2febf51d0b91b89f30cf8"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.465505 17903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:10:55.466169 17917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.466338 17903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:55.466442 17903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "6afbf5675df24dc08d71951a021d07b5"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "9d1e13a6f4beb7d4b7b6a22e9e208311"
server_key_iv: "68cdd66c37c2febf51d0b91b89f30cf8"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.466518 17903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:55.476090 17903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:55.476421 17903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:55.476574 17903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:55.480979 17903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34225
I20260927 21:10:55.481012 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34225 every 8 connection(s)
I20260927 21:10:55.481380 17903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:10:55.482331 17970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.484131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17903
I20260927 21:10:55.484246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:10:55.484445 392 external_mini_cluster.cc:1428] Setting key b734398cde949dfe9d9c8804b40aa93b
I20260927 21:10:55.484870 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5: Bootstrap starting.
I20260927 21:10:55.485574 17970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.485910 17970 log.cc:824] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.486717 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5: No bootstrap required, opened a new log
I20260927 21:10:55.489221 17970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afbf5675df24dc08d71951a021d07b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260927 21:10:55.489396 17970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.489460 17970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afbf5675df24dc08d71951a021d07b5, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.489593 17970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afbf5675df24dc08d71951a021d07b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260927 21:10:55.489681 17970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:10:55.489737 17970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:10:55.489799 17970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:55.490375 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.485009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.98.62:34225)
0927 21:10:55.485221 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.485231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.485307 (+ 76us) server_negotiation.cc:407] Connection header received
0927 21:10:55.485909 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.485918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.485997 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.486145 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.486376 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.487393 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.488493 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.488876 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.488959 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.489719 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.489723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.489723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:55.489943 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.489975 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:55.489984 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.490053 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.490132 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.490210 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260927 21:10:55.490397 17970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afbf5675df24dc08d71951a021d07b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260927 21:10:55.490530 17970 leader_election.cc:302] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6afbf5675df24dc08d71951a021d07b5; no voters:
I20260927 21:10:55.490700 17970 leader_election.cc:288] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:10:55.490733 17975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:55.490895 17975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 6afbf5675df24dc08d71951a021d07b5, State: Running, Role: LEADER
I20260927 21:10:55.490988 17970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:10:55.491052 17975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afbf5675df24dc08d71951a021d07b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } }
I20260927 21:10:55.491487 17976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6afbf5675df24dc08d71951a021d07b5. Latest consensus state: current_term: 1 leader_uuid: "6afbf5675df24dc08d71951a021d07b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afbf5675df24dc08d71951a021d07b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260927 21:10:55.491487 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6afbf5675df24dc08d71951a021d07b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afbf5675df24dc08d71951a021d07b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34225 } } }
I20260927 21:10:55.491638 17976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:55.491645 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:55.491962 17984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:10:55.492517 17984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:10:55.494063 17984 catalog_manager.cc:1347] Generated new cluster ID: fe4eb26570774d78ae0e62833c8d759c
I20260927 21:10:55.494108 17984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:10:55.525660 17984 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:10:55.526233 17984 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:10:55.529609 17984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5: Generated new TSK 0
I20260927 21:10:55.529799 17984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:10:55.541435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--builtin_ntp_servers=127.0.98.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:10:55.623808 17994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:55.624022 17994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:55.624043 17994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:55.625192 17994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:55.625231 17994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:55.625257 17994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:10:55.626623 17994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:55.626813 17994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:55.627074 17994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:55.631728 18002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.631739 17999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.631807 18000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.632015 17994 server_base.cc:1034] running on GCE node
I20260927 21:10:55.632337 17994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:55.632589 17994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:55.633751 17994 hybrid_clock.cc:648] HybridClock initialized: now 1790543455633743 us; error 39 us; skew 500 ppm
I20260927 21:10:55.634999 17994 webserver.cc:460] Webserver started at http://127.0.98.1:41033/ using document root <none> and password file <none>
I20260927 21:10:55.635264 17994 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:55.635332 17994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:55.635470 17994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:55.636603 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "14bc78ae035b486684be2f5ab941e22c"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "9491598cda426ecb0006cbb5c4773d5a"
server_key_iv: "f62fe200602e574670449a262662fb43"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.636965 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14bc78ae035b486684be2f5ab941e22c"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "9491598cda426ecb0006cbb5c4773d5a"
server_key_iv: "f62fe200602e574670449a262662fb43"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.638233 17994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:10:55.638914 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.639074 17994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:55.639215 17994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "14bc78ae035b486684be2f5ab941e22c"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "9491598cda426ecb0006cbb5c4773d5a"
server_key_iv: "f62fe200602e574670449a262662fb43"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.639297 17994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:55.650967 17994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:55.651278 17994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:55.651427 17994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:55.651646 17994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:55.652005 17994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:55.652048 17994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.652089 17994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:55.652115 17994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.657913 17994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42205
I20260927 21:10:55.657953 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42205 every 8 connection(s)
I20260927 21:10:55.658290 17994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:10:55.663336 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.659000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55301 (local address 127.0.98.62:34225)
0927 21:10:55.659103 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.659105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.659320 (+ 215us) server_negotiation.cc:407] Connection header received
0927 21:10:55.659777 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.659778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.659790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.659846 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.660481 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.660992 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.661974 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.662169 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.662232 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.663032 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.663033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.663034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:55.663038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.663053 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:55.663056 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.663172 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.663245 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.663271 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:10:55.663544 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.658970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.1:55301)
0927 21:10:55.659202 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.659236 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.659698 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.659912 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.659916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.659924 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:55.660407 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.660410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.661093 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.661094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.661864 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.661866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.662250 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.662251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.662357 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.662899 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:10:55.662899 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:55.662925 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:10:55.662947 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:55.663174 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:55.663177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:55.663179 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.663233 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.663310 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":116,"threads_started":1}
I20260927 21:10:55.664084 18122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260927 21:10:55.664182 18122 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:55.664376 18122 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260927 21:10:55.664892 17934 ts_manager.cc:194] Registered new tserver with Master: 14bc78ae035b486684be2f5ab941e22c (127.0.98.1:42205)
I20260927 21:10:55.665526 17934 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55301
I20260927 21:10:55.666515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 17994
I20260927 21:10:55.666594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:10:55.666762 392 external_mini_cluster.cc:1428] Setting key bebb73a6f06844e12a2ce19fee5d1770
I20260927 21:10:55.667585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--builtin_ntp_servers=127.0.98.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:10:55.748477 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:55.748665 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:55.748685 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:55.749797 18125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:55.749836 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:55.749884 18125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:10:55.751253 18125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:55.751484 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:55.751763 18125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:10:55.756757 18125 server_base.cc:1034] running on GCE node
W20260927 21:10:55.756721 18131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.756717 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.756739 18130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.757162 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:55.757357 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:55.758502 18125 hybrid_clock.cc:648] HybridClock initialized: now 1790543455758491 us; error 29 us; skew 500 ppm
I20260927 21:10:55.759732 18125 webserver.cc:460] Webserver started at http://127.0.98.2:38665/ using document root <none> and password file <none>
I20260927 21:10:55.759972 18125 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:55.760020 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:55.760178 18125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:55.761199 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "0d5a1c2d8bc540b59d2620620e233eb2"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "bac83fb5d2ac03d8eade2824c30e3eaa"
server_key_iv: "b92019145092e5454f6827bc71092b93"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.761565 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0d5a1c2d8bc540b59d2620620e233eb2"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "bac83fb5d2ac03d8eade2824c30e3eaa"
server_key_iv: "b92019145092e5454f6827bc71092b93"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.762809 18125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:10:55.763585 18139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.763765 18125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.763828 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "0d5a1c2d8bc540b59d2620620e233eb2"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "bac83fb5d2ac03d8eade2824c30e3eaa"
server_key_iv: "b92019145092e5454f6827bc71092b93"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.763914 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:55.780562 18125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:55.780859 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:55.781010 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:55.781235 18125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:55.781560 18125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:55.781615 18125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.781656 18125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:55.781680 18125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.787799 18125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38047
I20260927 21:10:55.787839 18252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38047 every 8 connection(s)
I20260927 21:10:55.788158 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:10:55.792039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18125
I20260927 21:10:55.792147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:10:55.792361 392 external_mini_cluster.cc:1428] Setting key 90e2159ff88629f2c0f4020ee9241480
I20260927 21:10:55.793371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--builtin_ntp_servers=127.0.98.20:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:10:55.793463 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.788846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41251 (local address 127.0.98.62:34225)
0927 21:10:55.788890 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.788892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.789138 (+ 246us) server_negotiation.cc:407] Connection header received
0927 21:10:55.789597 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.789598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.789607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.789661 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.790270 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.790815 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.791962 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.792232 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.792302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.793197 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.793199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.793200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:55.793204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.793219 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:55.793223 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.793265 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.793357 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.793386 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:10:55.793681 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.788805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.2:41251)
0927 21:10:55.789011 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.789050 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.789519 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.789720 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.789724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.789731 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:55.790207 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.790210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.790895 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.790896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.791886 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.791889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.792312 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.792313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.792481 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.793059 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:10:55.793059 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:55.793098 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:10:55.793126 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:55.793291 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:55.793295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:55.793298 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.793360 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.793461 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":116,"threads_started":1}
I20260927 21:10:55.794230 18253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260927 21:10:55.794329 18253 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:55.794516 18253 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260927 21:10:55.794978 17934 ts_manager.cc:194] Registered new tserver with Master: 0d5a1c2d8bc540b59d2620620e233eb2 (127.0.98.2:38047)
I20260927 21:10:55.795507 17934 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41251
W20260927 21:10:55.876474 18256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:55.876794 18256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:55.876828 18256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:55.878253 18256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:55.878314 18256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:55.878355 18256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:10:55.879746 18256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:55.879927 18256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:55.880168 18256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:55.884990 18262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.885038 18261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:55.884990 18264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:55.885217 18256 server_base.cc:1034] running on GCE node
I20260927 21:10:55.885434 18256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:55.885680 18256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:55.886847 18256 hybrid_clock.cc:648] HybridClock initialized: now 1790543455886833 us; error 27 us; skew 500 ppm
I20260927 21:10:55.888170 18256 webserver.cc:460] Webserver started at http://127.0.98.3:40661/ using document root <none> and password file <none>
I20260927 21:10:55.888410 18256 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:55.888458 18256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:55.888612 18256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:55.889686 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b0b7219bc2d4065897de5811a7b3405"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "b72b6e617fc3fe188543f2e2e6a6ee35"
server_key_iv: "0678ceb34efdaf730227df6aa5850e4a"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.890056 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b0b7219bc2d4065897de5811a7b3405"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "b72b6e617fc3fe188543f2e2e6a6ee35"
server_key_iv: "0678ceb34efdaf730227df6aa5850e4a"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.891646 18256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:10:55.892463 18270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.892620 18256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:55.892719 18256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "5b0b7219bc2d4065897de5811a7b3405"
format_stamp: "Formatted at 2026-09-27 21:10:55 on dist-test-slave-8bgs"
server_key: "b72b6e617fc3fe188543f2e2e6a6ee35"
server_key_iv: "0678ceb34efdaf730227df6aa5850e4a"
server_key_version: "encryptionkey@0"
I20260927 21:10:55.892807 18256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:55.907073 18256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:55.907442 18256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:55.907596 18256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:55.907826 18256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:55.908149 18256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:55.908205 18256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.908247 18256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:55.908274 18256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:55.913805 18256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40835
I20260927 21:10:55.913880 18383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40835 every 8 connection(s)
I20260927 21:10:55.914232 18256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:10:55.919118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18256
I20260927 21:10:55.919237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:10:55.919421 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.915145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39507 (local address 127.0.98.62:34225)
0927 21:10:55.915204 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.915207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.915442 (+ 235us) server_negotiation.cc:407] Connection header received
0927 21:10:55.915931 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.915933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.915942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.915993 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.916638 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.917120 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.918061 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.918236 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.918287 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.919119 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.919120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.919121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:55.919126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.919142 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:55.919147 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.919211 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.919314 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.919342 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:10:55.919446 392 external_mini_cluster.cc:1428] Setting key 9d01444b55e9d432af69d8c8cc8cc41f
I20260927 21:10:55.919677 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.915064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34225 (local address 127.0.98.3:39507)
0927 21:10:55.915310 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.915354 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.915846 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.916048 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.916052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.916062 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:55.916571 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.916575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.917200 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.917200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.917987 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.917990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.918310 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.918310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.918444 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.918954 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:10:55.918955 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:55.918982 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:10:55.919004 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:55.919222 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:55.919228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:55.919232 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.919311 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.919425 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260927 21:10:55.920161 18384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34225
I20260927 21:10:55.920341 18384 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:55.920545 18384 heartbeater.cc:511] Master 127.0.98.62:34225 requested a full tablet report, sending...
I20260927 21:10:55.920950 17934 ts_manager.cc:194] Registered new tserver with Master: 5b0b7219bc2d4065897de5811a7b3405 (127.0.98.3:40835)
I20260927 21:10:55.921275 17934 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39507
I20260927 21:10:55.931414 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:10:55.935683 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.932884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.0.98.62:34225)
0927 21:10:55.932956 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.932958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.933067 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:10:55.933107 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.933108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.933117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.933170 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:55.933310 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.933785 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.934488 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.934662 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.934710 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.935401 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:55.935402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:55.935402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:55.935407 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:55.935419 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:55.935422 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:55.935464 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.935518 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.935536 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:10:55.937513 17934 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:10:55.939150 17934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:10:55.945845 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.941575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.0.98.2:38047)
0927 21:10:55.942027 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.942030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.942040 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:10:55.942082 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.942084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.942100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.942199 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.942405 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.943456 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.944894 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.945448 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.945536 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.945632 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.945724 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.945761 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":115,"threads_started":1}
I20260927 21:10:55.945845 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.941453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38047 (local address 127.0.0.1:40298)
0927 21:10:55.941811 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.941855 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.941924 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.942239 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.942244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.942246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.942319 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.942321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.943549 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.943550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.944805 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.944807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.945548 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.945550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.945625 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.945638 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.945710 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.945761 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":78,"threads_started":1}
I20260927 21:10:55.946395 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.941453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40835 (local address 127.0.0.1:45834)
0927 21:10:55.941811 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.941855 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.941922 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.942489 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.942490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.942492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.942533 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.942535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.943877 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.943878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.944932 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.944934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.946156 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.946157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.946214 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.946217 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.946273 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.946316 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260927 21:10:55.946527 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.941803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.0.98.1:42205)
0927 21:10:55.942003 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.942008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.942677 (+ 669us) server_negotiation.cc:407] Connection header received
0927 21:10:55.942723 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.942725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.942740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.942825 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.942923 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.943979 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.945010 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.945740 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.945808 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.945892 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.946377 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.946429 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260927 21:10:55.946791 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.941814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45834 (local address 127.0.98.3:40835)
0927 21:10:55.942327 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:55.942332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:55.942342 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:10:55.942381 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:55.942383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:55.942396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:55.942469 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:55.942616 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.943787 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.945042 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.945740 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.945808 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.945892 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:55.946681 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 21:10:55.946724 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":71,"threads_started":1}
I20260927 21:10:55.946708 18187 tablet_service.cc:1467] Processing CreateTablet for tablet b35fdff941334a68b1e31bb039b8db50 (DEFAULT_TABLE table=test-table [id=7238c962064d4e2094ba285adccdc546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:55.946980 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:55.941684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42205 (local address 127.0.0.1:48916)
0927 21:10:55.942537 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:55.942545 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:55.942613 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:55.942825 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:55.942826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:55.942828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:55.942863 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.942864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.944083 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.944084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.944921 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:55.944924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:55.946147 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:55.946149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:55.946214 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:55.946219 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:10:55.946273 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:10:55.946316 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":68,"threads_started":1}
I20260927 21:10:55.947089 18187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35fdff941334a68b1e31bb039b8db50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.947657 18318 tablet_service.cc:1467] Processing CreateTablet for tablet b35fdff941334a68b1e31bb039b8db50 (DEFAULT_TABLE table=test-table [id=7238c962064d4e2094ba285adccdc546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:55.947976 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35fdff941334a68b1e31bb039b8db50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.947885 18056 tablet_service.cc:1467] Processing CreateTablet for tablet b35fdff941334a68b1e31bb039b8db50 (DEFAULT_TABLE table=test-table [id=7238c962064d4e2094ba285adccdc546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:55.948206 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35fdff941334a68b1e31bb039b8db50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:55.950419 18398 tablet_bootstrap.cc:492] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Bootstrap starting.
I20260927 21:10:55.951037 18398 tablet_bootstrap.cc:654] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.951305 18400 tablet_bootstrap.cc:492] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: Bootstrap starting.
I20260927 21:10:55.951428 18398 log.cc:824] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.951996 18400 tablet_bootstrap.cc:654] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.952194 18398 tablet_bootstrap.cc:492] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: No bootstrap required, opened a new log
I20260927 21:10:55.952266 18398 ts_tablet_manager.cc:1397] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:10:55.952304 18400 log.cc:824] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.953138 18400 tablet_bootstrap.cc:492] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: No bootstrap required, opened a new log
I20260927 21:10:55.953217 18400 ts_tablet_manager.cc:1397] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:10:55.954963 18398 raft_consensus.cc:348] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:55.951802 18399 tablet_bootstrap.cc:492] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Bootstrap starting.
I20260927 21:10:55.955156 18398 raft_consensus.cc:374] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.955228 18398 raft_consensus.cc:729] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0b7219bc2d4065897de5811a7b3405, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.955358 18398 consensus_queue.cc:260] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:55.955612 18398 ts_tablet_manager.cc:1428] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 21:10:55.955657 18384 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
I20260927 21:10:55.955803 18399 tablet_bootstrap.cc:654] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:55.955734 18400 raft_consensus.cc:348] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:55.955865 18400 raft_consensus.cc:374] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.955904 18400 raft_consensus.cc:729] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14bc78ae035b486684be2f5ab941e22c, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.956008 18400 consensus_queue.cc:260] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:55.956121 18399 log.cc:824] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:55.956280 18122 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
I20260927 21:10:55.956264 18400 ts_tablet_manager.cc:1428] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:10:55.956902 18399 tablet_bootstrap.cc:492] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: No bootstrap required, opened a new log
I20260927 21:10:55.956979 18399 ts_tablet_manager.cc:1397] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:10:55.958814 18399 raft_consensus.cc:348] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:55.958954 18399 raft_consensus.cc:374] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:55.958976 18399 raft_consensus.cc:729] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5a1c2d8bc540b59d2620620e233eb2, State: Initialized, Role: FOLLOWER
I20260927 21:10:55.959136 18399 consensus_queue.cc:260] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:55.959411 18399 ts_tablet_manager.cc:1428] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:10:55.959444 18253 heartbeater.cc:503] Master 127.0.98.62:34225 was elected leader, sending a full tablet report...
W20260927 21:10:56.038682 18254 tablet.cc:2369] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:10:56.124121 18403 raft_consensus.cc:482] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:56.124298 18403 raft_consensus.cc:504] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:56.124665 18403 leader_election.cc:288] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5a1c2d8bc540b59d2620620e233eb2 (127.0.98.2:38047), 14bc78ae035b486684be2f5ab941e22c (127.0.98.1:42205)
I20260927 21:10:56.127686 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:56.125121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37033 (local address 127.0.98.2:38047)
0927 21:10:56.125180 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:56.125184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:56.125197 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:10:56.125222 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:56.125222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:56.125230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:56.125298 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:56.125464 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.125956 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.126894 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.127337 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.127421 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:56.127431 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:56.127581 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 21:10:56.127621 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:10:56.127724 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:56.124954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38047 (local address 127.0.98.3:37033)
0927 21:10:56.125037 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:56.125051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:56.125106 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:56.125302 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:56.125303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:56.125304 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:56.125379 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:56.125381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.126043 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.126044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:56.126794 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:56.126796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.127426 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.127427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:56.127496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:56.127500 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:56.127576 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:10:56.127618 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 21:10:56.127735 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:56.125152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55993 (local address 127.0.98.1:42205)
0927 21:10:56.125200 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:56.125202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:56.125311 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:10:56.125388 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:56.125389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:56.125403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:56.125457 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:56.125621 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.126102 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.127009 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.127386 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.127471 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:56.127480 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:56.127632 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:10:56.127667 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:10:56.127831 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:56.125066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42205 (local address 127.0.98.3:55993)
0927 21:10:56.125230 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:56.125239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:56.125332 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:56.125472 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:56.125473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:56.125474 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:56.125548 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:56.125549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.126193 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.126194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:56.126918 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:56.126920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.127471 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.127472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:56.127555 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:56.127558 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:56.127626 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:10:56.127664 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260927 21:10:56.128175 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35fdff941334a68b1e31bb039b8db50" candidate_uuid: "5b0b7219bc2d4065897de5811a7b3405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14bc78ae035b486684be2f5ab941e22c" is_pre_election: true
I20260927 21:10:56.128191 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35fdff941334a68b1e31bb039b8db50" candidate_uuid: "5b0b7219bc2d4065897de5811a7b3405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" is_pre_election: true
I20260927 21:10:56.128345 18076 raft_consensus.cc:2454] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b0b7219bc2d4065897de5811a7b3405 in term 0.
I20260927 21:10:56.128348 18207 raft_consensus.cc:2454] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b0b7219bc2d4065897de5811a7b3405 in term 0.
I20260927 21:10:56.128566 18273 leader_election.cc:302] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5a1c2d8bc540b59d2620620e233eb2, 5b0b7219bc2d4065897de5811a7b3405; no voters:
I20260927 21:10:56.128726 18403 raft_consensus.cc:2781] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:10:56.128791 18403 raft_consensus.cc:482] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:10:56.128808 18403 raft_consensus.cc:3037] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:56.129525 18403 raft_consensus.cc:504] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:56.129626 18403 leader_election.cc:288] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [CANDIDATE]: Term 1 election: Requested vote from peers 0d5a1c2d8bc540b59d2620620e233eb2 (127.0.98.2:38047), 14bc78ae035b486684be2f5ab941e22c (127.0.98.1:42205)
I20260927 21:10:56.129802 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35fdff941334a68b1e31bb039b8db50" candidate_uuid: "5b0b7219bc2d4065897de5811a7b3405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5a1c2d8bc540b59d2620620e233eb2"
I20260927 21:10:56.129863 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35fdff941334a68b1e31bb039b8db50" candidate_uuid: "5b0b7219bc2d4065897de5811a7b3405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14bc78ae035b486684be2f5ab941e22c"
I20260927 21:10:56.129896 18207 raft_consensus.cc:3037] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:56.129933 18076 raft_consensus.cc:3037] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:56.130843 18076 raft_consensus.cc:2454] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b0b7219bc2d4065897de5811a7b3405 in term 1.
I20260927 21:10:56.130869 18207 raft_consensus.cc:2454] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b0b7219bc2d4065897de5811a7b3405 in term 1.
I20260927 21:10:56.131125 18273 leader_election.cc:302] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5a1c2d8bc540b59d2620620e233eb2, 5b0b7219bc2d4065897de5811a7b3405; no voters:
I20260927 21:10:56.131223 18403 raft_consensus.cc:2781] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:56.131381 18403 raft_consensus.cc:686] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 1 LEADER]: Becoming Leader. State: Replica: 5b0b7219bc2d4065897de5811a7b3405, State: Running, Role: LEADER
I20260927 21:10:56.131489 18403 consensus_queue.cc:237] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } }
I20260927 21:10:56.132361 17934 catalog_manager.cc:5495] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b0b7219bc2d4065897de5811a7b3405 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5b0b7219bc2d4065897de5811a7b3405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0b7219bc2d4065897de5811a7b3405" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 } health_report { overall_health: UNKNOWN } } }
W20260927 21:10:56.158986 18123 tablet.cc:2369] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:10:56.164762 18385 tablet.cc:2369] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:10:56.173017 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:56.170694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.3:40835)
0927 21:10:56.170756 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:56.170758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:56.170767 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:10:56.170806 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:56.170807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:56.170844 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:56.170915 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:10:56.171052 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.171556 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.172301 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:56.172487 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:56.172544 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:56.172671 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:10:56.172766 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:10:56.172826 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:56.172862 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:10:56.172878 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:10:56.174806 18076 raft_consensus.cc:1261] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 1 FOLLOWER]: Refusing update from remote peer 5b0b7219bc2d4065897de5811a7b3405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:56.174829 18207 raft_consensus.cc:1261] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 1 FOLLOWER]: Refusing update from remote peer 5b0b7219bc2d4065897de5811a7b3405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:56.175215 18403 consensus_queue.cc:1035] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14bc78ae035b486684be2f5ab941e22c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:56.175354 18408 consensus_queue.cc:1035] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5a1c2d8bc540b59d2620620e233eb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:56.177928 18412 mvcc.cc:205] Tried to move back new op lower bound from 7334065996489150464 to 7334065996315185152. Current Snapshot: MvccSnapshot[applied={T|T < 7334065996489150464}]
I20260927 21:10:56.179853 17934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:56.179943 17934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:56.180887 17934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6afbf5675df24dc08d71951a021d07b5: Sending DeleteTablet for 3 replicas of tablet b35fdff941334a68b1e31bb039b8db50
I20260927 21:10:56.181178 18318 tablet_service.cc:1514] Processing DeleteTablet for tablet b35fdff941334a68b1e31bb039b8db50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:56 UTC) from {username='slave'} at 127.0.0.1:45834
I20260927 21:10:56.181244 18187 tablet_service.cc:1514] Processing DeleteTablet for tablet b35fdff941334a68b1e31bb039b8db50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:56 UTC) from {username='slave'} at 127.0.0.1:40298
I20260927 21:10:56.181277 18056 tablet_service.cc:1514] Processing DeleteTablet for tablet b35fdff941334a68b1e31bb039b8db50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:56 UTC) from {username='slave'} at 127.0.0.1:48916
I20260927 21:10:56.181459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17994
I20260927 21:10:56.181495 18421 tablet_replica.cc:330] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: stopping tablet replica
I20260927 21:10:56.181557 18423 tablet_replica.cc:330] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c: stopping tablet replica
I20260927 21:10:56.181588 18421 raft_consensus.cc:2229] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:10:56.181648 18423 raft_consensus.cc:2229] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:56.181674 18421 raft_consensus.cc:2258] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:56.181738 18423 raft_consensus.cc:2258] T b35fdff941334a68b1e31bb039b8db50 P 14bc78ae035b486684be2f5ab941e22c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:56.181799 18422 tablet_replica.cc:330] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: stopping tablet replica
I20260927 21:10:56.182037 18422 raft_consensus.cc:2229] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:56.182130 18422 raft_consensus.cc:2258] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:56.182565 18422 ts_tablet_manager.cc:1905] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:56.183238 18421 ts_tablet_manager.cc:1905] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:56.184713 18422 ts_tablet_manager.cc:1918] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:56.184777 18422 log.cc:1196] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/b35fdff941334a68b1e31bb039b8db50
I20260927 21:10:56.184777 18421 ts_tablet_manager.cc:1918] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:56.184854 18421 log.cc:1196] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/b35fdff941334a68b1e31bb039b8db50
I20260927 21:10:56.185058 18422 ts_tablet_manager.cc:1939] T b35fdff941334a68b1e31bb039b8db50 P 0d5a1c2d8bc540b59d2620620e233eb2: Deleting consensus metadata
I20260927 21:10:56.185123 18421 ts_tablet_manager.cc:1939] T b35fdff941334a68b1e31bb039b8db50 P 5b0b7219bc2d4065897de5811a7b3405: Deleting consensus metadata
I20260927 21:10:56.185472 17920 catalog_manager.cc:4841] TS 0d5a1c2d8bc540b59d2620620e233eb2 (127.0.98.2:38047): tablet b35fdff941334a68b1e31bb039b8db50 (table test-table [id=7238c962064d4e2094ba285adccdc546]) successfully deleted
I20260927 21:10:56.185717 17921 catalog_manager.cc:4841] TS 5b0b7219bc2d4065897de5811a7b3405 (127.0.98.3:40835): tablet b35fdff941334a68b1e31bb039b8db50 (table test-table [id=7238c962064d4e2094ba285adccdc546]) successfully deleted
W20260927 21:10:56.189136 17920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42205 (error 108)
I20260927 21:10:56.189340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18125
I20260927 21:10:56.190393 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:56.190275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42205 (local address 127.0.0.1:48932)
0927 21:10:56.190316 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:56.190365 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 21:10:56.190532 17920 catalog_manager.cc:4584] TS 14bc78ae035b486684be2f5ab941e22c (127.0.98.1:42205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b35fdff941334a68b1e31bb039b8db50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42205: connect: Connection refused (error 111)
I20260927 21:10:56.194185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18256
I20260927 21:10:56.198971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 17903
2026-09-27T21:10:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (854 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:10:56 dist-test-slave-8bgs krb5kdc[18427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:10:56 dist-test-slave-8bgs krb5kdc[18427](info): set up 2 sockets
Sep 27 21:10:56 dist-test-slave-8bgs krb5kdc[18427](info): commencing operation
krb5kdc: starting...
W20260927 21:10:57.230801 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:10:57.342188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:57.422065 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:57.422276 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:57.422298 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:57.423501 18440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:10:57.423542 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:57.423557 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:57.423609 18440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:10:57.423625 18440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:10:57.425114 18440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:57.425365 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:57.425598 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:57.430460 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.430500 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.430505 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:57.430934 18440 server_base.cc:1034] running on GCE node
I20260927 21:10:57.431236 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:57.431474 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:57.432614 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790543457432596 us; error 39 us; skew 500 ppm
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:57.434921 18440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:10:57.435393 18440 webserver.cc:460] Webserver started at http://127.0.98.62:37613/ using document root <none> and password file <none>
I20260927 21:10:57.435621 18440 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:57.435673 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:57.435832 18440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:57.436869 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "c8e1541ae5524b4e8205db3333ef2b03"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "629c04c648c99dea1de8653c2c16aa75"
server_key_iv: "85091c6fd3d6d2e76e63f826b4defa0e"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.437230 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8e1541ae5524b4e8205db3333ef2b03"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "629c04c648c99dea1de8653c2c16aa75"
server_key_iv: "85091c6fd3d6d2e76e63f826b4defa0e"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.438508 18440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:10:57.439256 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.439430 18440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:10:57.439518 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "c8e1541ae5524b4e8205db3333ef2b03"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "629c04c648c99dea1de8653c2c16aa75"
server_key_iv: "85091c6fd3d6d2e76e63f826b4defa0e"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.439604 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:57.452287 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:57.453995 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:57.454142 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:57.458170 18440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42605
I20260927 21:10:57.458194 18507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42605 every 8 connection(s)
I20260927 21:10:57.458530 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:10:57.459417 18508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:57.461555 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03: Bootstrap starting.
I20260927 21:10:57.462067 18508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:57.462292 18508 log.cc:824] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:57.462919 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03: No bootstrap required, opened a new log
I20260927 21:10:57.464490 18508 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e1541ae5524b4e8205db3333ef2b03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42605 } }
I20260927 21:10:57.464622 18508 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:57.464643 18508 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e1541ae5524b4e8205db3333ef2b03, State: Initialized, Role: FOLLOWER
I20260927 21:10:57.464771 18508 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e1541ae5524b4e8205db3333ef2b03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42605 } }
I20260927 21:10:57.464854 18508 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:10:57.464876 18508 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:10:57.464919 18508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:57.465397 18508 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e1541ae5524b4e8205db3333ef2b03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42605 } }
I20260927 21:10:57.465487 18508 leader_election.cc:302] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e1541ae5524b4e8205db3333ef2b03; no voters:
I20260927 21:10:57.465606 18508 leader_election.cc:288] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:10:57.465653 18511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:57.465828 18511 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [term 1 LEADER]: Becoming Leader. State: Replica: c8e1541ae5524b4e8205db3333ef2b03, State: Running, Role: LEADER
I20260927 21:10:57.465865 18508 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:10:57.465939 18511 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e1541ae5524b4e8205db3333ef2b03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42605 } }
I20260927 21:10:57.466466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18440
I20260927 21:10:57.466466 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e1541ae5524b4e8205db3333ef2b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e1541ae5524b4e8205db3333ef2b03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42605 } } }
I20260927 21:10:57.466544 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:57.466578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:10:57.466794 392 external_mini_cluster.cc:1428] Setting key 48b62eec62e3b7c037c24f16063c805f
I20260927 21:10:57.467064 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e1541ae5524b4e8205db3333ef2b03. Latest consensus state: current_term: 1 leader_uuid: "c8e1541ae5524b4e8205db3333ef2b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e1541ae5524b4e8205db3333ef2b03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42605 } } }
I20260927 21:10:57.467239 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03 [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:57.467545 18520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:10:57.468230 18520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:10:57.469856 18520 catalog_manager.cc:1347] Generated new cluster ID: 380cc26e08ad4646ae9abcf086d1d88e
I20260927 21:10:57.469916 18520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:57.478971 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.467315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.0.98.62:42605)
0927 21:10:57.467730 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.467739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.467777 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:10:57.467901 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.467906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.467947 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.468095 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:57.468964 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.470299 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.471456 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.472102 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.472219 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.475196 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:57.475200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:57.475201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:57.475213 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:57.477273 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.477685 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.477689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.477689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.477796 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.478013 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.478014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.478014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.478325 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:57.478488 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.478589 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.478732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":98,"threads_started":1}
I20260927 21:10:57.479450 18520 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:10:57.480062 18520 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:10:57.484354 18520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03: Generated new TSK 0
I20260927 21:10:57.484545 18520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:10:57.521993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42605
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:57.601490 18534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:57.601696 18534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:57.601719 18534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:57.602882 18534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:57.602924 18534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:57.602950 18534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:10:57.604432 18534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:57.604676 18534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:57.604900 18534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:57.609715 18540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.609721 18539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.609728 18542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:57.609933 18534 server_base.cc:1034] running on GCE node
I20260927 21:10:57.610167 18534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:57.610388 18534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:57.611534 18534 hybrid_clock.cc:648] HybridClock initialized: now 1790543457611520 us; error 28 us; skew 500 ppm
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:57.613863 18534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:10:57.614308 18534 webserver.cc:460] Webserver started at http://127.0.98.1:37491/ using document root <none> and password file <none>
I20260927 21:10:57.614528 18534 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:57.614595 18534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:57.614734 18534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:57.615783 18534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "a259a2e0162c40fb8d78359c537d8e3b"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "d0fad5fba509bdaafbc710cccf27a939"
server_key_iv: "bea9441c09ac55eee3cb2a30feda5fb7"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.616159 18534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a259a2e0162c40fb8d78359c537d8e3b"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "d0fad5fba509bdaafbc710cccf27a939"
server_key_iv: "bea9441c09ac55eee3cb2a30feda5fb7"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.617422 18534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:10:57.618043 18549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.618239 18534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:57.618330 18534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "a259a2e0162c40fb8d78359c537d8e3b"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "d0fad5fba509bdaafbc710cccf27a939"
server_key_iv: "bea9441c09ac55eee3cb2a30feda5fb7"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.618400 18534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:57.646111 18534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:57.647959 18534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:57.648125 18534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:57.648352 18534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:57.648658 18534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:57.648715 18534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.648749 18534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:57.648775 18534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.654310 18534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40639
I20260927 21:10:57.654349 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40639 every 8 connection(s)
I20260927 21:10:57.654661 18534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:10:57.656590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18534
I20260927 21:10:57.656733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:10:57.656955 392 external_mini_cluster.cc:1428] Setting key fad0ffd18f239780d1ed3ae6e50d8313
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 21:10:57.664589 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.655370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42965 (local address 127.0.98.62:42605)
0927 21:10:57.655412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.655414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.655642 (+ 228us) server_negotiation.cc:407] Connection header received
0927 21:10:57.656803 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.656804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.656821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.656877 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:57.657702 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.658198 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.659531 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.659745 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.659829 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.661884 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:57.661885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:57.661886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:57.661893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:57.663253 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.663638 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.663640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.663641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.663705 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.663945 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.663946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.663946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.664060 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:57.664131 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.664413 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.664502 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:10:57.664758 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.655348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42605 (local address 127.0.98.1:42965)
0927 21:10:57.655519 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:57.655552 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:57.656687 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:57.656971 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:57.656975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:57.657339 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:57.657617 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.657624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.658288 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.658289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.659434 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.659437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.659838 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.659840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.659964 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.660460 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:57.660468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:57.661791 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:57.663329 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:57.663336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:57.663338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:57.663552 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:57.663779 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:57.663779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:57.663780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:57.663880 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:57.664135 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:57.664141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:57.664273 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 21:10:57.664375 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:10:57.664534 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260927 21:10:57.665333 18663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42605
I20260927 21:10:57.665434 18663 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:57.665599 18663 heartbeater.cc:511] Master 127.0.98.62:42605 requested a full tablet report, sending...
I20260927 21:10:57.666118 18472 ts_manager.cc:194] Registered new tserver with Master: a259a2e0162c40fb8d78359c537d8e3b (127.0.98.1:40639)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 21:10:57.666882 18472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:10:57.679986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42605
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:57.759619 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:57.759807 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:57.759829 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:57.760993 18668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:57.761032 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:57.761091 18668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:10:57.762545 18668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:57.762791 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:57.763005 18668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:57.768019 18676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.768057 18673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.768010 18674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:57.768257 18668 server_base.cc:1034] running on GCE node
I20260927 21:10:57.768481 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:57.768735 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:57.769897 18668 hybrid_clock.cc:648] HybridClock initialized: now 1790543457769944 us; error 88 us; skew 500 ppm
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:57.772295 18668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:10:57.772832 18668 webserver.cc:460] Webserver started at http://127.0.98.2:46183/ using document root <none> and password file <none>
I20260927 21:10:57.773079 18668 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:57.773151 18668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:57.773301 18668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:57.774350 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "aad5247de6d14726917f3c21a31538ad"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "0d94e305b3972823db46e743e9b037ec"
server_key_iv: "7fe13a3bad1b7fe947a9331242633893"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.774710 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aad5247de6d14726917f3c21a31538ad"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "0d94e305b3972823db46e743e9b037ec"
server_key_iv: "7fe13a3bad1b7fe947a9331242633893"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.776180 18668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:10:57.776880 18683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.777047 18668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:57.777151 18668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "aad5247de6d14726917f3c21a31538ad"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "0d94e305b3972823db46e743e9b037ec"
server_key_iv: "7fe13a3bad1b7fe947a9331242633893"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.777228 18668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:57.789470 18668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:57.791473 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:57.791635 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:57.791870 18668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:57.792176 18668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:57.792236 18668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.792279 18668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:57.792307 18668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.797958 18668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32817
I20260927 21:10:57.798005 18796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32817 every 8 connection(s)
I20260927 21:10:57.798357 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:10:57.804175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18668
I20260927 21:10:57.804291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:57.804509 392 external_mini_cluster.cc:1428] Setting key 27bec92f99bd0209f16ccd69c39a1dc6
I20260927 21:10:57.807868 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.799042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42605 (local address 127.0.98.2:38217)
0927 21:10:57.799242 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:57.799283 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:57.800196 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:57.800479 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:57.800483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:57.800747 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:57.800937 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.800940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.801602 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.801602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.802397 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.802399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.802765 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.802765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.802857 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.803313 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:57.803321 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:57.804783 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:57.806574 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:57.806579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:57.806581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:57.806787 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:57.806993 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:57.806993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:57.806994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:57.807074 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:57.807390 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:57.807394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:57.807508 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 21:10:57.807591 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:10:57.807706 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260927 21:10:57.807897 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.799138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38217 (local address 127.0.98.62:42605)
0927 21:10:57.799197 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.799200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.799375 (+ 175us) server_negotiation.cc:407] Connection header received
0927 21:10:57.800325 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.800326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.800336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.800401 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:57.801017 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.801522 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.802492 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.802675 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.802749 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.804909 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:57.804910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:57.804911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:57.804918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:57.806486 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.806862 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.806863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.806863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.806930 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.807170 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.807172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.807172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.807321 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:57.807395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.807644 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.807746 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:10:57.808660 18797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42605
I20260927 21:10:57.808769 18797 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:57.808974 18797 heartbeater.cc:511] Master 127.0.98.62:42605 requested a full tablet report, sending...
I20260927 21:10:57.809343 18472 ts_manager.cc:194] Registered new tserver with Master: aad5247de6d14726917f3c21a31538ad (127.0.98.2:32817)
I20260927 21:10:57.809738 18472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38217
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:10:57.829195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42605
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:10:57.909320 18802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:57.909551 18802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:57.909590 18802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:57.910739 18802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:57.910804 18802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:57.910856 18802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:10:57.912437 18802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:57.912698 18802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:57.912943 18802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:57.917965 18810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.917989 18808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:57.917981 18807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:57.918011 18802 server_base.cc:1034] running on GCE node
I20260927 21:10:57.918478 18802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:57.918737 18802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:57.919902 18802 hybrid_clock.cc:648] HybridClock initialized: now 1790543457919899 us; error 36 us; skew 500 ppm
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:10:57.922453 18802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:10:57.922902 18802 webserver.cc:460] Webserver started at http://127.0.98.3:46493/ using document root <none> and password file <none>
I20260927 21:10:57.923167 18802 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:57.923233 18802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:57.923386 18802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:57.924412 18802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "41c3f4e16ad3461f9ca590bdb257f92f"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "baf39d6a311eca5c5cb53c704f1d4393"
server_key_iv: "b905f5a6cc905b9005b30ea9599088a9"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.924777 18802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "41c3f4e16ad3461f9ca590bdb257f92f"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "baf39d6a311eca5c5cb53c704f1d4393"
server_key_iv: "b905f5a6cc905b9005b30ea9599088a9"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.926064 18802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:10:57.926731 18817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.926904 18802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:10:57.926999 18802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "41c3f4e16ad3461f9ca590bdb257f92f"
format_stamp: "Formatted at 2026-09-27 21:10:57 on dist-test-slave-8bgs"
server_key: "baf39d6a311eca5c5cb53c704f1d4393"
server_key_iv: "b905f5a6cc905b9005b30ea9599088a9"
server_key_version: "encryptionkey@0"
I20260927 21:10:57.927084 18802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:57.947712 18802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:57.950030 18802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:57.950201 18802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:57.950431 18802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:57.950745 18802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:57.950800 18802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.950843 18802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:57.950870 18802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:57.956802 18802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45401
I20260927 21:10:57.956888 18930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45401 every 8 connection(s)
I20260927 21:10:57.957221 18802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:10:57 dist-test-slave-8bgs krb5kdc[18427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:10:57.964002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18802
I20260927 21:10:57.964129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:10:57.964372 392 external_mini_cluster.cc:1428] Setting key 90d9b7401b34e076769f165a653769b9
I20260927 21:10:57.966466 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.957916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53353 (local address 127.0.98.62:42605)
0927 21:10:57.957965 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.957968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.958189 (+ 221us) server_negotiation.cc:407] Connection header received
0927 21:10:57.959081 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.959082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.959156 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.959217 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:57.959782 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.960265 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.961200 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.961372 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.961420 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.963372 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:57.963374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:57.963374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:57.963381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:57.965015 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.965472 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.965473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.965474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.965540 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.965798 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.965799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.965800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.965920 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:57.966036 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.966298 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.966378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:10:57.966663 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.957902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42605 (local address 127.0.98.3:53353)
0927 21:10:57.958066 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:57.958101 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:57.958993 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:57.959273 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:57.959276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:57.959516 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:57.959707 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.959711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.960340 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.960341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.961129 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.961131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.961441 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.961442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.961533 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.961944 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:10:57.961952 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:57.963303 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:57.965091 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:57.965099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:57.965101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:57.965382 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:57.965605 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:10:57.965606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:10:57.965606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:10:57.965724 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:10:57.966014 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:57.966021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:57.966165 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:10:57.966276 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:10:57.966431 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260927 21:10:57.967119 18931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42605
I20260927 21:10:57.967216 18931 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:57.967361 18931 heartbeater.cc:511] Master 127.0.98.62:42605 requested a full tablet report, sending...
I20260927 21:10:57.967724 18472 ts_manager.cc:194] Registered new tserver with Master: 41c3f4e16ad3461f9ca590bdb257f92f (127.0.98.3:45401)
I20260927 21:10:57.968109 18472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53353
I20260927 21:10:57.975932 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:10:57.983763 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.977291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.0.98.62:42605)
0927 21:10:57.977346 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.977348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.977461 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:10:57.977537 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.977538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.977548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.977612 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:57.978208 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.978681 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.979650 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.979904 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.979986 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.981348 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:57.981350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:57.981350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:10:57.981358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:57.982538 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.982920 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.982922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.982922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.982982 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:10:57.983202 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:10:57.983203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:10:57.983203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:10:57.983356 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:57.983474 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.983596 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.983661 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:10:57.985419 18472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:10:57.987376 18472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:10:57.993870 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.989470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40639 (local address 127.0.0.1:56632)
0927 21:10:57.989754 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:57.989795 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:57.989870 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:57.990110 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:57.990113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:57.990115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:57.990181 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.990182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.991342 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.991343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.992692 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.992704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.993571 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.993571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.993627 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.993637 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:10:57.993740 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:10:57.993796 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":33,"thread_start_us":46,"threads_started":1}
I20260927 21:10:57.993880 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.989585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.0.98.1:40639)
0927 21:10:57.989791 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.989794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.989872 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:10:57.990003 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.990006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.990021 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.990092 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:57.990347 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.991247 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.992813 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.993491 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.993572 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.993631 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.993761 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.993807 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260927 21:10:57.994240 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.989667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32817 (local address 127.0.0.1:50396)
0927 21:10:57.990171 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:57.990179 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:57.990261 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:57.990524 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:57.990524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:57.990526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:57.990589 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.990590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.991778 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.991779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.992678 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.992681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.994041 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.994042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.994102 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.994105 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:57.994151 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:10:57.994186 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260927 21:10:57.994724 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.989570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.98.3:45401)
0927 21:10:57.989739 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.989744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.989909 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:10:57.990009 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.990011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.990029 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.990081 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:57.990303 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.991410 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.993661 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.994284 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.994351 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.994399 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.994599 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.994641 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260927 21:10:57.994699 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce02b68f889470b94ed6456b1ecc275 (DEFAULT_TABLE table=test-table [id=17332f01e1b14356b92738e972e6000c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:57.994910 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.989486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45401 (local address 127.0.0.1:35106)
0927 21:10:57.989833 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:57.989859 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:57.989922 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:57.990181 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:57.990182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:57.990184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:57.990223 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.990224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.991511 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.991512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.992705 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:57.992707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.994432 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.994433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:57.994504 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.994507 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:57.994802 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 21:10:57.994846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260927 21:10:57.995078 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce02b68f889470b94ed6456b1ecc275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:57.995549 18865 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce02b68f889470b94ed6456b1ecc275 (DEFAULT_TABLE table=test-table [id=17332f01e1b14356b92738e972e6000c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:57.995867 18865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce02b68f889470b94ed6456b1ecc275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:57.997769 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:57.989866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.0.98.2:32817)
0927 21:10:57.990027 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:57.990031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:57.990317 (+ 286us) server_negotiation.cc:407] Connection header received
0927 21:10:57.990359 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:57.990361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:57.990376 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:57.990443 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:57.990672 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.991679 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.992759 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:57.993491 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:57.993572 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:57.993631 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:57.997647 (+ 4016us) server_negotiation.cc:299] Negotiation successful
0927 21:10:57.997695 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260927 21:10:57.997977 18948 tablet_bootstrap.cc:492] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: Bootstrap starting.
I20260927 21:10:57.998436 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce02b68f889470b94ed6456b1ecc275 (DEFAULT_TABLE table=test-table [id=17332f01e1b14356b92738e972e6000c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:57.998600 18948 tablet_bootstrap.cc:654] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:57.998652 18949 tablet_bootstrap.cc:492] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Bootstrap starting.
I20260927 21:10:57.998759 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce02b68f889470b94ed6456b1ecc275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:57.999223 18948 log.cc:824] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:57.999366 18949 tablet_bootstrap.cc:654] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:57.999897 18949 log.cc:824] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:58.000030 18948 tablet_bootstrap.cc:492] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: No bootstrap required, opened a new log
I20260927 21:10:58.000108 18948 ts_tablet_manager.cc:1397] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:10:58.000576 18949 tablet_bootstrap.cc:492] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: No bootstrap required, opened a new log
I20260927 21:10:58.000660 18949 ts_tablet_manager.cc:1397] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:10:58.001861 18952 tablet_bootstrap.cc:492] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: Bootstrap starting.
I20260927 21:10:58.002144 18948 raft_consensus.cc:348] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.002305 18948 raft_consensus.cc:374] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.002347 18948 raft_consensus.cc:729] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a259a2e0162c40fb8d78359c537d8e3b, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.002451 18948 consensus_queue.cc:260] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.002502 18952 tablet_bootstrap.cc:654] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:58.002568 18949 raft_consensus.cc:348] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.002741 18663 heartbeater.cc:503] Master 127.0.98.62:42605 was elected leader, sending a full tablet report...
I20260927 21:10:58.002749 18949 raft_consensus.cc:374] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.002761 18948 ts_tablet_manager.cc:1428] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:10:58.002789 18949 raft_consensus.cc:729] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c3f4e16ad3461f9ca590bdb257f92f, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.002931 18949 consensus_queue.cc:260] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.003075 18952 log.cc:824] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:58.003372 18931 heartbeater.cc:503] Master 127.0.98.62:42605 was elected leader, sending a full tablet report...
I20260927 21:10:58.003381 18949 ts_tablet_manager.cc:1428] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:10:58.003942 18952 tablet_bootstrap.cc:492] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: No bootstrap required, opened a new log
I20260927 21:10:58.004043 18952 ts_tablet_manager.cc:1397] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:10:58.005636 18952 raft_consensus.cc:348] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.005776 18952 raft_consensus.cc:374] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.005833 18952 raft_consensus.cc:729] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad5247de6d14726917f3c21a31538ad, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.005937 18952 consensus_queue.cc:260] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.006206 18952 ts_tablet_manager.cc:1428] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:10:58.006210 18797 heartbeater.cc:503] Master 127.0.98.62:42605 was elected leader, sending a full tablet report...
I20260927 21:10:58.038913 18953 raft_consensus.cc:482] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:58.039024 18953 raft_consensus.cc:504] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.039357 18953 leader_election.cc:288] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c3f4e16ad3461f9ca590bdb257f92f (127.0.98.3:45401), aad5247de6d14726917f3c21a31538ad (127.0.98.2:32817)
I20260927 21:10:58.043457 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.039640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58493 (local address 127.0.98.3:45401)
0927 21:10:58.039691 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.039694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.039711 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:10:58.039844 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.039845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.039853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.039934 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.040141 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.040906 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.042459 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.043039 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.043138 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.043150 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.043345 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.043388 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:10:58.043457 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.039727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47395 (local address 127.0.98.2:32817)
0927 21:10:58.039800 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.039801 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.039904 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:10:58.039931 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.039932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.039944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.039995 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.040141 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.040905 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.042459 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.043036 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.043138 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.043149 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.043345 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.043385 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:10:58.043479 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.039541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45401 (local address 127.0.98.1:58493)
0927 21:10:58.039630 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.039644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.039704 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.039950 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.039950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.039952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.040029 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.040032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.040990 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.040990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.042366 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.042369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.043160 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.043162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.043256 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.043260 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.043328 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.043382 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":41}
I20260927 21:10:58.043485 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.039657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32817 (local address 127.0.98.1:47395)
0927 21:10:58.039815 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.039824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.039884 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.040003 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.040004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.040005 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.040072 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.040074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.040990 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.040990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.042366 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.042369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.043173 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.043174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.043256 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.043260 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.043328 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.043382 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":117,"thread_start_us":113,"threads_started":1}
I20260927 21:10:58.043843 18885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce02b68f889470b94ed6456b1ecc275" candidate_uuid: "a259a2e0162c40fb8d78359c537d8e3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" is_pre_election: true
I20260927 21:10:58.044013 18885 raft_consensus.cc:2454] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a259a2e0162c40fb8d78359c537d8e3b in term 0.
I20260927 21:10:58.043973 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce02b68f889470b94ed6456b1ecc275" candidate_uuid: "a259a2e0162c40fb8d78359c537d8e3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad5247de6d14726917f3c21a31538ad" is_pre_election: true
I20260927 21:10:58.044106 18751 raft_consensus.cc:2454] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a259a2e0162c40fb8d78359c537d8e3b in term 0.
I20260927 21:10:58.044271 18553 leader_election.cc:302] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c3f4e16ad3461f9ca590bdb257f92f, a259a2e0162c40fb8d78359c537d8e3b; no voters:
I20260927 21:10:58.044420 18953 raft_consensus.cc:2781] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:10:58.044510 18953 raft_consensus.cc:482] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:10:58.044549 18953 raft_consensus.cc:3037] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.045258 18953 raft_consensus.cc:504] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.045399 18953 leader_election.cc:288] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [CANDIDATE]: Term 1 election: Requested vote from peers 41c3f4e16ad3461f9ca590bdb257f92f (127.0.98.3:45401), aad5247de6d14726917f3c21a31538ad (127.0.98.2:32817)
I20260927 21:10:58.045575 18885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce02b68f889470b94ed6456b1ecc275" candidate_uuid: "a259a2e0162c40fb8d78359c537d8e3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c3f4e16ad3461f9ca590bdb257f92f"
I20260927 21:10:58.045586 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce02b68f889470b94ed6456b1ecc275" candidate_uuid: "a259a2e0162c40fb8d78359c537d8e3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad5247de6d14726917f3c21a31538ad"
I20260927 21:10:58.045671 18751 raft_consensus.cc:3037] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.045676 18885 raft_consensus.cc:3037] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.046363 18885 raft_consensus.cc:2454] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a259a2e0162c40fb8d78359c537d8e3b in term 1.
I20260927 21:10:58.046422 18751 raft_consensus.cc:2454] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a259a2e0162c40fb8d78359c537d8e3b in term 1.
I20260927 21:10:58.046574 18553 leader_election.cc:302] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c3f4e16ad3461f9ca590bdb257f92f, a259a2e0162c40fb8d78359c537d8e3b; no voters:
I20260927 21:10:58.046686 18953 raft_consensus.cc:2781] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:58.046880 18953 raft_consensus.cc:686] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 1 LEADER]: Becoming Leader. State: Replica: a259a2e0162c40fb8d78359c537d8e3b, State: Running, Role: LEADER
I20260927 21:10:58.047026 18953 consensus_queue.cc:237] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } }
I20260927 21:10:58.047884 18471 catalog_manager.cc:5495] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b reported cstate change: term changed from 0 to 1, leader changed from <none> to a259a2e0162c40fb8d78359c537d8e3b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a259a2e0162c40fb8d78359c537d8e3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a259a2e0162c40fb8d78359c537d8e3b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 } health_report { overall_health: UNKNOWN } } }
W20260927 21:10:58.048873 18798 tablet.cc:2369] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:10:58.066634 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.064118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.1:40639)
0927 21:10:58.064174 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.064176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.064185 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:10:58.064237 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.064238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.064255 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.064327 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:10:58.064454 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.064921 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.065797 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.065993 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.066096 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.066268 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:10:58.066383 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:10:58.066507 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.066552 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.066563 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:10:58.068598 18751 raft_consensus.cc:1261] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 1 FOLLOWER]: Refusing update from remote peer a259a2e0162c40fb8d78359c537d8e3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:58.068612 18885 raft_consensus.cc:1261] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 1 FOLLOWER]: Refusing update from remote peer a259a2e0162c40fb8d78359c537d8e3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:58.069017 18958 consensus_queue.cc:1035] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad5247de6d14726917f3c21a31538ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:58.069151 18953 consensus_queue.cc:1035] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c3f4e16ad3461f9ca590bdb257f92f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:58.071550 18961 mvcc.cc:205] Tried to move back new op lower bound from 7334066004245983232 to 7334066004161200128. Current Snapshot: MvccSnapshot[applied={T|T < 7334066004245983232}]
I20260927 21:10:58.071581 18963 mvcc.cc:205] Tried to move back new op lower bound from 7334066004245983232 to 7334066004161200128. Current Snapshot: MvccSnapshot[applied={T|T < 7334066004245983232}]
I20260927 21:10:58.073836 18471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:58.073932 18471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:58.074749 18471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8e1541ae5524b4e8205db3333ef2b03: Sending DeleteTablet for 3 replicas of tablet 9ce02b68f889470b94ed6456b1ecc275
I20260927 21:10:58.075060 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce02b68f889470b94ed6456b1ecc275 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56632
I20260927 21:10:58.075179 18865 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce02b68f889470b94ed6456b1ecc275 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35106
I20260927 21:10:58.075218 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce02b68f889470b94ed6456b1ecc275 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50396
I20260927 21:10:58.075333 18970 tablet_replica.cc:330] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b: stopping tablet replica
I20260927 21:10:58.075436 18971 tablet_replica.cc:330] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: stopping tablet replica
I20260927 21:10:58.075485 18970 raft_consensus.cc:2229] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:10:58.075574 18971 raft_consensus.cc:2229] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:58.075645 18970 raft_consensus.cc:2258] T 9ce02b68f889470b94ed6456b1ecc275 P a259a2e0162c40fb8d78359c537d8e3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:58.075678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18534
I20260927 21:10:58.075729 18971 raft_consensus.cc:2258] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:58.076794 18971 ts_tablet_manager.cc:1905] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:58.078601 18971 ts_tablet_manager.cc:1918] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:58.078687 18971 log.cc:1196] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/9ce02b68f889470b94ed6456b1ecc275
I20260927 21:10:58.078954 18971 ts_tablet_manager.cc:1939] T 9ce02b68f889470b94ed6456b1ecc275 P 41c3f4e16ad3461f9ca590bdb257f92f: Deleting consensus metadata
I20260927 21:10:58.079422 18459 catalog_manager.cc:4841] TS 41c3f4e16ad3461f9ca590bdb257f92f (127.0.98.3:45401): tablet 9ce02b68f889470b94ed6456b1ecc275 (table test-table [id=17332f01e1b14356b92738e972e6000c]) successfully deleted
W20260927 21:10:58.080931 18456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40639 (error 108)
I20260927 21:10:58.081069 18972 tablet_replica.cc:330] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad: stopping tablet replica
I20260927 21:10:58.081177 18972 raft_consensus.cc:2229] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:58.081290 18972 raft_consensus.cc:2258] T 9ce02b68f889470b94ed6456b1ecc275 P aad5247de6d14726917f3c21a31538ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:58.081406 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.081286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40639 (local address 127.0.0.1:56646)
0927 21:10:58.081326 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.081370 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 21:10:58.081466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18668
W20260927 21:10:58.082805 18456 catalog_manager.cc:4584] TS a259a2e0162c40fb8d78359c537d8e3b (127.0.98.1:40639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ce02b68f889470b94ed6456b1ecc275: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40639: connect: Connection refused (error 111)
I20260927 21:10:58.086498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18802
I20260927 21:10:58.086522 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.086414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32817 (local address 127.0.0.1:50410)
0927 21:10:58.086464 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.086489 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 21:10:58.090713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18440
2026-09-27T21:10:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1892 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T21:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:10:58Z Disabled control of system clock
I20260927 21:10:58.117923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37313
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:10:58.197310 18977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:58.197516 18977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:58.197536 18977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:58.198683 18977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:10:58.198737 18977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:58.198750 18977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:58.198763 18977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:10:58.198814 18977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:10:58.200172 18977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:58.200429 18977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:58.200721 18977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:58.205413 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.205422 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:58.205464 18977 server_base.cc:1034] running on GCE node
W20260927 21:10:58.205426 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:58.205853 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:58.206123 18977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:58.207263 18977 hybrid_clock.cc:648] HybridClock initialized: now 1790543458207243 us; error 36 us; skew 500 ppm
I20260927 21:10:58.208423 18977 webserver.cc:460] Webserver started at http://127.0.98.62:46713/ using document root <none> and password file <none>
I20260927 21:10:58.208657 18977 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:58.208702 18977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:58.208847 18977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:58.209878 18977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "22794287b3df4b9f9d3ed77c18406380"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "a4c21d2856d8ac3ff14cddae22633f19"
server_key_iv: "22530659c32fd2a6972a4d19012676b6"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.210227 18977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "22794287b3df4b9f9d3ed77c18406380"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "a4c21d2856d8ac3ff14cddae22633f19"
server_key_iv: "22530659c32fd2a6972a4d19012676b6"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.211683 18977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:10:58.212423 18991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.212599 18977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:10:58.212699 18977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "22794287b3df4b9f9d3ed77c18406380"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "a4c21d2856d8ac3ff14cddae22633f19"
server_key_iv: "22530659c32fd2a6972a4d19012676b6"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.212779 18977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:58.233052 18977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:58.233361 18977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:58.233510 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:58.237239 18977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37313
I20260927 21:10:58.237346 19043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37313 every 8 connection(s)
I20260927 21:10:58.237638 18977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:10:58.238603 19044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:58.240922 19044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: Bootstrap starting.
I20260927 21:10:58.241503 19044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:58.241796 19044 log.cc:824] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:58.242095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 18977
I20260927 21:10:58.242218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:10:58.242457 392 external_mini_cluster.cc:1428] Setting key 8ee837027cf28615db66f78408491533
I20260927 21:10:58.242632 19044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: No bootstrap required, opened a new log
I20260927 21:10:58.244451 19044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22794287b3df4b9f9d3ed77c18406380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260927 21:10:58.244606 19044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.244635 19044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22794287b3df4b9f9d3ed77c18406380, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.244776 19044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22794287b3df4b9f9d3ed77c18406380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260927 21:10:58.244865 19044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:10:58.244889 19044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:10:58.244956 19044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.245486 19044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22794287b3df4b9f9d3ed77c18406380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260927 21:10:58.245564 19044 leader_election.cc:302] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22794287b3df4b9f9d3ed77c18406380; no voters:
I20260927 21:10:58.245750 19044 leader_election.cc:288] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:10:58.245775 19049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:58.245898 19049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [term 1 LEADER]: Becoming Leader. State: Replica: 22794287b3df4b9f9d3ed77c18406380, State: Running, Role: LEADER
I20260927 21:10:58.245992 19044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:10:58.246009 19049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22794287b3df4b9f9d3ed77c18406380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260927 21:10:58.246363 19049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22794287b3df4b9f9d3ed77c18406380. Latest consensus state: current_term: 1 leader_uuid: "22794287b3df4b9f9d3ed77c18406380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22794287b3df4b9f9d3ed77c18406380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } } }
I20260927 21:10:58.246452 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22794287b3df4b9f9d3ed77c18406380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22794287b3df4b9f9d3ed77c18406380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } } }
I20260927 21:10:58.246470 19049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [sys.catalog]: This master's current role is: LEADER
I20260927 21:10:58.246551 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380 [sys.catalog]: This master's current role is: LEADER
W20260927 21:10:58.247849 19065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:10:58.247957 19065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:10:58.248068 19056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:10:58.248234 19056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:10:58.249163 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.242995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.98.62:37313)
0927 21:10:58.243232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.243240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.243269 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:10:58.243839 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.243846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.243901 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.244042 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:58.244258 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.245772 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.246851 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.247642 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.247778 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.248481 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:58.248484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:58.248485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:58.248685 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:58.248728 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:58.248742 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:58.248810 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.248881 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.248958 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
I20260927 21:10:58.249739 19056 catalog_manager.cc:1347] Generated new cluster ID: d90a15943f8c4d53937530761cf89841
I20260927 21:10:58.249794 19056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:10:58.257495 19056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:10:58.257953 19056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:10:58.267215 19056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: Generated new TSK 0
I20260927 21:10:58.267374 19056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:10:58.268894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:10:58.357590 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:58.357786 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:58.357808 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:58.358963 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:58.359004 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:58.359031 19068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:10:58.360689 19068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:58.360929 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:58.361244 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:58.366922 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.366950 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.367065 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:58.367305 19068 server_base.cc:1034] running on GCE node
I20260927 21:10:58.367578 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:58.367835 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:58.369001 19068 hybrid_clock.cc:648] HybridClock initialized: now 1790543458368990 us; error 26 us; skew 500 ppm
I20260927 21:10:58.370242 19068 webserver.cc:460] Webserver started at http://127.0.98.1:41785/ using document root <none> and password file <none>
I20260927 21:10:58.370512 19068 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:58.370592 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:58.370738 19068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:58.371848 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "467045ecb8104c5eaaa40f4b7e2e5750"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "0ff9905fd35a0a124b9ca411d0ba079c"
server_key_iv: "4c5ad105193e7c14c563f1312777ecc8"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.372206 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "467045ecb8104c5eaaa40f4b7e2e5750"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "0ff9905fd35a0a124b9ca411d0ba079c"
server_key_iv: "4c5ad105193e7c14c563f1312777ecc8"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.373544 19068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:10:58.374342 19082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.374605 19068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:58.374724 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "467045ecb8104c5eaaa40f4b7e2e5750"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "0ff9905fd35a0a124b9ca411d0ba079c"
server_key_iv: "4c5ad105193e7c14c563f1312777ecc8"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.374817 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:58.409169 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:58.409457 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:58.409562 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:58.409759 19068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:58.410140 19068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:58.410176 19068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.410225 19068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:58.410260 19068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.415992 19068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42759
I20260927 21:10:58.416095 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42759 every 8 connection(s)
I20260927 21:10:58.416429 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:10:58.421401 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.417193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58479 (local address 127.0.98.62:37313)
0927 21:10:58.417252 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.417254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.417492 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:10:58.417946 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.417947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.417959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.418015 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:58.418627 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.419128 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.420099 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.420291 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.420358 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.421187 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:58.421189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:58.421189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:58.421193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:58.421208 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:58.421212 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:58.421259 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.421319 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.421343 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:10:58.421631 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.417197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37313 (local address 127.0.98.1:58479)
0927 21:10:58.417363 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.417397 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.417864 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.418065 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.418069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.418076 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:58.418560 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.418563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.419199 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.419200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.419970 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.419973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.420376 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.420377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.420487 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.421048 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:10:58.421048 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:58.421077 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:10:58.421099 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:58.421258 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:58.421260 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:58.421263 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.421307 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.421384 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 21:10:58.422214 19196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37313
I20260927 21:10:58.422323 19196 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:58.422502 19196 heartbeater.cc:511] Master 127.0.98.62:37313 requested a full tablet report, sending...
I20260927 21:10:58.423075 19008 ts_manager.cc:194] Registered new tserver with Master: 467045ecb8104c5eaaa40f4b7e2e5750 (127.0.98.1:42759)
I20260927 21:10:58.423807 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58479
I20260927 21:10:58.424808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19068
I20260927 21:10:58.424925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:10:58.425135 392 external_mini_cluster.cc:1428] Setting key 25d3ba75f970203861b68e3bfa902db6
I20260927 21:10:58.426028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:10:58.508287 19199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:58.508474 19199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:58.508493 19199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:58.509541 19199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:58.509573 19199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:58.509598 19199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:10:58.510926 19199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:58.511175 19199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:58.511466 19199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:58.516204 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.516249 19205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.516355 19207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:58.516553 19199 server_base.cc:1034] running on GCE node
I20260927 21:10:58.516764 19199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:58.517023 19199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:58.518183 19199 hybrid_clock.cc:648] HybridClock initialized: now 1790543458518178 us; error 38 us; skew 500 ppm
I20260927 21:10:58.519464 19199 webserver.cc:460] Webserver started at http://127.0.98.2:42189/ using document root <none> and password file <none>
I20260927 21:10:58.519701 19199 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:58.519770 19199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:58.519899 19199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:58.520932 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "43c7d264c12a442ca215aa9f1d19afc5"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "0799b30e5e0aea2974bf4bec91a75756"
server_key_iv: "ae9671dac528e59fea22b2f913480ca6"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.521301 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43c7d264c12a442ca215aa9f1d19afc5"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "0799b30e5e0aea2974bf4bec91a75756"
server_key_iv: "ae9671dac528e59fea22b2f913480ca6"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.522557 19199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:10:58.523370 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.523540 19199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:10:58.523644 19199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "43c7d264c12a442ca215aa9f1d19afc5"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "0799b30e5e0aea2974bf4bec91a75756"
server_key_iv: "ae9671dac528e59fea22b2f913480ca6"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.523720 19199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:58.542335 19199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:58.542639 19199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:58.542793 19199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:58.543025 19199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:58.543360 19199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:58.543416 19199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.543454 19199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:58.543483 19199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.549242 19199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36305
I20260927 21:10:58.549305 19326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36305 every 8 connection(s)
I20260927 21:10:58.549628 19199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:10:58.550601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19199
I20260927 21:10:58.550709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:10:58.550904 392 external_mini_cluster.cc:1428] Setting key 2db399247420c0035e9561c6bb8d7d7c
I20260927 21:10:58.551844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:10:58.555135 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.550384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50273 (local address 127.0.98.62:37313)
0927 21:10:58.550462 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.550464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.550657 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:10:58.551364 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.551365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.551377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.551432 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:58.552229 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.552748 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.553728 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.553938 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.554009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.554867 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:58.554868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:58.554869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:58.554874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:58.554889 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:58.554893 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:58.554940 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.555009 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.555036 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:10:58.555342 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.550350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37313 (local address 127.0.98.2:50273)
0927 21:10:58.550525 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.550565 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.551262 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.551502 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.551506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.551516 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:58.552141 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.552146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.552854 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.552855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.553630 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.553632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.554027 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.554028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.554171 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.554726 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:10:58.554727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:58.554754 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:10:58.554776 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:58.554940 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:58.554945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:58.554948 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.555014 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.555130 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260927 21:10:58.555828 19327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37313
I20260927 21:10:58.555925 19327 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:58.556087 19327 heartbeater.cc:511] Master 127.0.98.62:37313 requested a full tablet report, sending...
I20260927 21:10:58.556480 19008 ts_manager.cc:194] Registered new tserver with Master: 43c7d264c12a442ca215aa9f1d19afc5 (127.0.98.2:36305)
I20260927 21:10:58.557016 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50273
W20260927 21:10:58.632081 19330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:10:58.632278 19330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:10:58.632300 19330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:10:58.633344 19330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:10:58.633376 19330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:10:58.633417 19330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:10:58.634701 19330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:10:58.634879 19330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:10:58.635165 19330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:10:58.639707 19336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.639698 19335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:10:58.639698 19338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:10:58.639926 19330 server_base.cc:1034] running on GCE node
I20260927 21:10:58.640143 19330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:10:58.640409 19330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:10:58.641566 19330 hybrid_clock.cc:648] HybridClock initialized: now 1790543458641548 us; error 30 us; skew 500 ppm
I20260927 21:10:58.642733 19330 webserver.cc:460] Webserver started at http://127.0.98.3:36987/ using document root <none> and password file <none>
I20260927 21:10:58.642976 19330 fs_manager.cc:359] Metadata directory not provided
I20260927 21:10:58.643045 19330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:10:58.643204 19330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:10:58.644227 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "e08fe5a3d787451dbbecd1277ee9b7af"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "032864f111789304df1aacd31122d7c8"
server_key_iv: "25131003aceac69f164d55b3198b1c8c"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.644601 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e08fe5a3d787451dbbecd1277ee9b7af"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "032864f111789304df1aacd31122d7c8"
server_key_iv: "25131003aceac69f164d55b3198b1c8c"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.645946 19330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:10:58.646734 19344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.646903 19330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:10:58.646997 19330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "e08fe5a3d787451dbbecd1277ee9b7af"
format_stamp: "Formatted at 2026-09-27 21:10:58 on dist-test-slave-8bgs"
server_key: "032864f111789304df1aacd31122d7c8"
server_key_iv: "25131003aceac69f164d55b3198b1c8c"
server_key_version: "encryptionkey@0"
I20260927 21:10:58.647168 19330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:10:58.665222 19330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:10:58.665450 19330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:10:58.665546 19330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:10:58.665695 19330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:10:58.666015 19330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:10:58.666055 19330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.666076 19330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:10:58.666126 19330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:10:58.671941 19330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43835
I20260927 21:10:58.671991 19457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43835 every 8 connection(s)
I20260927 21:10:58.672313 19330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:10:58.676497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19330
I20260927 21:10:58.676710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:10:58.676923 392 external_mini_cluster.cc:1428] Setting key 29024edb3b52b92ef53086f93b08fde2
I20260927 21:10:58.677549 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.672980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60693 (local address 127.0.98.62:37313)
0927 21:10:58.673048 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.673050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.673288 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:10:58.673827 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.673829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.673840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.673886 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:58.674560 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.675061 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.676094 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.676293 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.676364 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.677225 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:58.677227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:58.677227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:58.677232 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:58.677247 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:58.677251 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:58.677310 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.677423 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.677448 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:10:58.677735 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.672954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37313 (local address 127.0.98.3:60693)
0927 21:10:58.673165 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.673199 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.673703 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.673955 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.673960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.673969 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:10:58.674464 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.674469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.675196 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.675197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.675987 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.675989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.676383 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.676384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.676519 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.677069 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:10:58.677070 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:10:58.677098 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:10:58.677119 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:10:58.677324 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:10:58.677331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:10:58.677334 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.677412 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.677524 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260927 21:10:58.678166 19458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37313
I20260927 21:10:58.678241 19458 heartbeater.cc:464] Registering TS with master...
I20260927 21:10:58.678426 19458 heartbeater.cc:511] Master 127.0.98.62:37313 requested a full tablet report, sending...
I20260927 21:10:58.678766 19008 ts_manager.cc:194] Registered new tserver with Master: e08fe5a3d787451dbbecd1277ee9b7af (127.0.98.3:43835)
I20260927 21:10:58.679122 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60693
I20260927 21:10:58.688834 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:10:58.692953 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.690080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.0.98.62:37313)
0927 21:10:58.690118 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.690121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.690248 (+ 127us) server_negotiation.cc:407] Connection header received
0927 21:10:58.690310 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.690311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.690321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.690390 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:10:58.690563 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.691020 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.691780 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.691971 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.692040 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.692671 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:10:58.692673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:10:58.692673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:10:58.692677 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:10:58.692695 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:10:58.692700 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:10:58.692790 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.692861 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.692870 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:10:58.694330 19008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:10:58.695892 19008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:10:58.701994 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.697896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36305 (local address 127.0.0.1:55260)
0927 21:10:58.698055 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.698064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.698121 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.698465 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.698468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.698470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.698556 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.698558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.699919 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.699919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.700900 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.700902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.701765 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.701766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.701817 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.701827 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.701872 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.701905 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":54,"spinlock_wait_cycles":384,"thread_start_us":99,"threads_started":1}
I20260927 21:10:58.702013 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.698143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.0.98.2:36305)
0927 21:10:58.698298 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.698302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.698313 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:10:58.698347 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.698349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.698362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.698502 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.698675 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.699841 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.700991 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.701683 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.701761 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.701851 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.701886 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.701926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260927 21:10:58.702273 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.697895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43835 (local address 127.0.0.1:55362)
0927 21:10:58.698013 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.698054 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.698121 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.698465 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.698468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.698473 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.698550 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.698552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.699997 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.699998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.701137 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.701139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.701981 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.701981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.702037 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.702041 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.702152 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.702195 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":34,"thread_start_us":99,"threads_started":1}
I20260927 21:10:58.702533 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.698118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.0.98.3:43835)
0927 21:10:58.698283 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.698288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.698298 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:10:58.698336 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.698338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.698354 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.698629 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.698696 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.699837 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.701223 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.701901 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.701968 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.702042 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.702421 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.702457 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260927 21:10:58.702972 19261 tablet_service.cc:1467] Processing CreateTablet for tablet c31d2557345549cd8b91b4567d64a9bb (DEFAULT_TABLE table=test-table [id=cd32f58461da4acda8c771e8ef703c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:58.703153 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.697895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42759 (local address 127.0.0.1:39974)
0927 21:10:58.698487 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.698496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.698554 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.698828 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.698829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.698830 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.698862 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.698864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.699795 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.699796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.701137 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.701139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.702893 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.702894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.702956 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.702960 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.703019 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.703058 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":52,"thread_start_us":66,"threads_started":1}
I20260927 21:10:58.703189 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.698311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.0.98.1:42759)
0927 21:10:58.698462 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.698467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.698613 (+ 146us) server_negotiation.cc:407] Connection header received
0927 21:10:58.698669 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.698671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.698686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.698761 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.698943 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.699702 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.702083 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.702812 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.702884 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.702968 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.703029 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.703070 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260927 21:10:58.703200 19392 tablet_service.cc:1467] Processing CreateTablet for tablet c31d2557345549cd8b91b4567d64a9bb (DEFAULT_TABLE table=test-table [id=cd32f58461da4acda8c771e8ef703c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:58.703383 19261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31d2557345549cd8b91b4567d64a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:58.703536 19392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31d2557345549cd8b91b4567d64a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:58.703806 19130 tablet_service.cc:1467] Processing CreateTablet for tablet c31d2557345549cd8b91b4567d64a9bb (DEFAULT_TABLE table=test-table [id=cd32f58461da4acda8c771e8ef703c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:10:58.704098 19130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31d2557345549cd8b91b4567d64a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:10:58.705792 19472 tablet_bootstrap.cc:492] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Bootstrap starting.
I20260927 21:10:58.706465 19472 tablet_bootstrap.cc:654] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:58.706864 19472 log.cc:824] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:58.706898 19473 tablet_bootstrap.cc:492] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Bootstrap starting.
I20260927 21:10:58.707543 19473 tablet_bootstrap.cc:654] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:58.707561 19472 tablet_bootstrap.cc:492] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: No bootstrap required, opened a new log
I20260927 21:10:58.707638 19472 ts_tablet_manager.cc:1397] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:10:58.707831 19473 log.cc:824] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:58.708470 19473 tablet_bootstrap.cc:492] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: No bootstrap required, opened a new log
I20260927 21:10:58.708519 19473 ts_tablet_manager.cc:1397] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:10:58.709846 19472 raft_consensus.cc:348] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.710037 19472 raft_consensus.cc:374] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.710084 19472 raft_consensus.cc:729] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c7d264c12a442ca215aa9f1d19afc5, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.710197 19472 consensus_queue.cc:260] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.710465 19472 ts_tablet_manager.cc:1428] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:10:58.710510 19327 heartbeater.cc:503] Master 127.0.98.62:37313 was elected leader, sending a full tablet report...
I20260927 21:10:58.707278 19475 tablet_bootstrap.cc:492] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: Bootstrap starting.
I20260927 21:10:58.710583 19473 raft_consensus.cc:348] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.710728 19473 raft_consensus.cc:374] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.710763 19473 raft_consensus.cc:729] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08fe5a3d787451dbbecd1277ee9b7af, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.710877 19473 consensus_queue.cc:260] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.711169 19458 heartbeater.cc:503] Master 127.0.98.62:37313 was elected leader, sending a full tablet report...
I20260927 21:10:58.711197 19475 tablet_bootstrap.cc:654] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: Neither blocks nor log segments found. Creating new log.
I20260927 21:10:58.711148 19473 ts_tablet_manager.cc:1428] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:10:58.711519 19475 log.cc:824] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: Log is configured to *not* fsync() on all Append() calls
I20260927 21:10:58.712288 19475 tablet_bootstrap.cc:492] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: No bootstrap required, opened a new log
I20260927 21:10:58.712356 19475 ts_tablet_manager.cc:1397] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 21:10:58.714166 19475 raft_consensus.cc:348] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.714282 19475 raft_consensus.cc:374] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:10:58.714301 19475 raft_consensus.cc:729] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467045ecb8104c5eaaa40f4b7e2e5750, State: Initialized, Role: FOLLOWER
I20260927 21:10:58.714430 19475 consensus_queue.cc:260] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.714684 19475 ts_tablet_manager.cc:1428] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 21:10:58.714706 19196 heartbeater.cc:503] Master 127.0.98.62:37313 was elected leader, sending a full tablet report...
W20260927 21:10:58.800176 19328 tablet.cc:2369] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:10:58.805173 19480 raft_consensus.cc:482] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:10:58.805294 19480 raft_consensus.cc:504] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.805668 19480 leader_election.cc:288] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c7d264c12a442ca215aa9f1d19afc5 (127.0.98.2:36305), e08fe5a3d787451dbbecd1277ee9b7af (127.0.98.3:43835)
I20260927 21:10:58.808610 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.805860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36305 (local address 127.0.98.1:44071)
0927 21:10:58.805898 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.805909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.805951 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.806141 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.806142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.806144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.806236 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.806238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.806964 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.806965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.807782 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.807784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.808351 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.808352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.808436 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.808439 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.808492 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.808523 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 21:10:58.808629 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.805929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44071 (local address 127.0.98.2:36305)
0927 21:10:58.805985 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.805987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.805998 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:10:58.806028 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.806029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.806038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.806120 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.806308 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.806869 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.807875 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.808274 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.808331 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.808338 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.808513 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.808563 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:10:58.808713 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.805947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35985 (local address 127.0.98.3:43835)
0927 21:10:58.805996 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.805998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.806271 (+ 273us) server_negotiation.cc:407] Connection header received
0927 21:10:58.806303 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.806303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.806312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.806357 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:10:58.806488 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.806956 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.807967 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.808345 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.808414 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.808424 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.808593 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.808642 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:10:58.808733 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.805871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43835 (local address 127.0.98.1:35985)
0927 21:10:58.806159 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.806170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:10:58.806239 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:10:58.806362 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:10:58.806362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:10:58.806364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:10:58.806425 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.806427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.807045 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.807045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.807883 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:10:58.807885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.808432 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.808433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:10:58.808515 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.808520 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:10:58.808576 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:10:58.808609 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":138,"thread_start_us":97,"threads_started":1}
I20260927 21:10:58.809101 19412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31d2557345549cd8b91b4567d64a9bb" candidate_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" is_pre_election: true
I20260927 21:10:58.809204 19281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31d2557345549cd8b91b4567d64a9bb" candidate_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7d264c12a442ca215aa9f1d19afc5" is_pre_election: true
I20260927 21:10:58.809252 19412 raft_consensus.cc:2454] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467045ecb8104c5eaaa40f4b7e2e5750 in term 0.
I20260927 21:10:58.809371 19281 raft_consensus.cc:2454] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467045ecb8104c5eaaa40f4b7e2e5750 in term 0.
I20260927 21:10:58.809504 19083 leader_election.cc:302] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467045ecb8104c5eaaa40f4b7e2e5750, e08fe5a3d787451dbbecd1277ee9b7af; no voters:
I20260927 21:10:58.809638 19480 raft_consensus.cc:2781] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:10:58.809705 19480 raft_consensus.cc:482] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:10:58.809748 19480 raft_consensus.cc:3037] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.810410 19480 raft_consensus.cc:504] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.810545 19480 leader_election.cc:288] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [CANDIDATE]: Term 1 election: Requested vote from peers 43c7d264c12a442ca215aa9f1d19afc5 (127.0.98.2:36305), e08fe5a3d787451dbbecd1277ee9b7af (127.0.98.3:43835)
I20260927 21:10:58.810695 19281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31d2557345549cd8b91b4567d64a9bb" candidate_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7d264c12a442ca215aa9f1d19afc5"
I20260927 21:10:58.810693 19412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31d2557345549cd8b91b4567d64a9bb" candidate_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08fe5a3d787451dbbecd1277ee9b7af"
I20260927 21:10:58.810765 19281 raft_consensus.cc:3037] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.810796 19412 raft_consensus.cc:3037] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:10:58.811510 19281 raft_consensus.cc:2454] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467045ecb8104c5eaaa40f4b7e2e5750 in term 1.
I20260927 21:10:58.811677 19085 leader_election.cc:302] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c7d264c12a442ca215aa9f1d19afc5, 467045ecb8104c5eaaa40f4b7e2e5750; no voters:
I20260927 21:10:58.811748 19412 raft_consensus.cc:2454] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467045ecb8104c5eaaa40f4b7e2e5750 in term 1.
I20260927 21:10:58.811793 19480 raft_consensus.cc:2781] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:10:58.811954 19480 raft_consensus.cc:686] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 1 LEADER]: Becoming Leader. State: Replica: 467045ecb8104c5eaaa40f4b7e2e5750, State: Running, Role: LEADER
I20260927 21:10:58.812071 19480 consensus_queue.cc:237] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } }
I20260927 21:10:58.812893 19007 catalog_manager.cc:5495] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 reported cstate change: term changed from 0 to 1, leader changed from <none> to 467045ecb8104c5eaaa40f4b7e2e5750 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "467045ecb8104c5eaaa40f4b7e2e5750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42759 } health_report { overall_health: HEALTHY } } }
I20260927 21:10:58.819862 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.817550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.0.98.1:42759)
0927 21:10:58.817596 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:10:58.817598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:10:58.817608 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:10:58.817679 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:10:58.817680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:10:58.817705 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:10:58.817778 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:10:58.817894 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.818341 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.819080 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:10:58.819291 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:10:58.819388 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:10:58.819538 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:10:58.819628 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:10:58.819731 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:10:58.819762 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 21:10:58.819770 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:10:58.821521 19281 raft_consensus.cc:1261] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 1 FOLLOWER]: Refusing update from remote peer 467045ecb8104c5eaaa40f4b7e2e5750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:58.821585 19412 raft_consensus.cc:1261] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 1 FOLLOWER]: Refusing update from remote peer 467045ecb8104c5eaaa40f4b7e2e5750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:10:58.821882 19482 consensus_queue.cc:1035] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c7d264c12a442ca215aa9f1d19afc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:58.821990 19480 consensus_queue.cc:1035] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08fe5a3d787451dbbecd1277ee9b7af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:10:58.824646 19485 mvcc.cc:205] Tried to move back new op lower bound from 7334066007330054144 to 7334066007294705664. Current Snapshot: MvccSnapshot[applied={T|T < 7334066007330054144}]
I20260927 21:10:58.825321 19487 mvcc.cc:205] Tried to move back new op lower bound from 7334066007330054144 to 7334066007294705664. Current Snapshot: MvccSnapshot[applied={T|T < 7334066007330054144}]
I20260927 21:10:58.825421 19486 mvcc.cc:205] Tried to move back new op lower bound from 7334066007330054144 to 7334066007294705664. Current Snapshot: MvccSnapshot[applied={T|T < 7334066007330054144 or (T in {7334066007330054144})}]
I20260927 21:10:58.827343 19008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:58.827440 19008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:10:58.828120 19008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22794287b3df4b9f9d3ed77c18406380: Sending DeleteTablet for 3 replicas of tablet c31d2557345549cd8b91b4567d64a9bb
I20260927 21:10:58.828434 19261 tablet_service.cc:1514] Processing DeleteTablet for tablet c31d2557345549cd8b91b4567d64a9bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:58 UTC) from {username='slave'} at 127.0.0.1:55260
I20260927 21:10:58.828439 19130 tablet_service.cc:1514] Processing DeleteTablet for tablet c31d2557345549cd8b91b4567d64a9bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:58 UTC) from {username='slave'} at 127.0.0.1:39974
I20260927 21:10:58.828580 19392 tablet_service.cc:1514] Processing DeleteTablet for tablet c31d2557345549cd8b91b4567d64a9bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:10:58 UTC) from {username='slave'} at 127.0.0.1:55362
I20260927 21:10:58.828770 19497 tablet_replica.cc:330] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: stopping tablet replica
I20260927 21:10:58.828859 19496 tablet_replica.cc:330] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750: stopping tablet replica
I20260927 21:10:58.828922 19497 raft_consensus.cc:2229] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:58.829062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19068
I20260927 21:10:58.829074 19497 raft_consensus.cc:2258] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:58.828975 19496 raft_consensus.cc:2229] T c31d2557345549cd8b91b4567d64a9bb P 467045ecb8104c5eaaa40f4b7e2e5750 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:10:58.829778 19495 tablet_replica.cc:330] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: stopping tablet replica
I20260927 21:10:58.829872 19495 raft_consensus.cc:2229] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:10:58.829969 19495 raft_consensus.cc:2258] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:10:58.830447 19495 ts_tablet_manager.cc:1905] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:58.831832 19495 ts_tablet_manager.cc:1918] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:58.831888 19495 log.cc:1196] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/c31d2557345549cd8b91b4567d64a9bb
I20260927 21:10:58.832132 19495 ts_tablet_manager.cc:1939] T c31d2557345549cd8b91b4567d64a9bb P 43c7d264c12a442ca215aa9f1d19afc5: Deleting consensus metadata
I20260927 21:10:58.832557 18994 catalog_manager.cc:4841] TS 43c7d264c12a442ca215aa9f1d19afc5 (127.0.98.2:36305): tablet c31d2557345549cd8b91b4567d64a9bb (table test-table [id=cd32f58461da4acda8c771e8ef703c64]) successfully deleted
W20260927 21:10:58.834452 18995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42759 (error 108)
I20260927 21:10:58.834757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19199
I20260927 21:10:58.835353 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:10:58.834847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42759 (local address 127.0.0.1:39986)
0927 21:10:58.835278 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 21:10:58.835327 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411}
W20260927 21:10:58.835472 18995 catalog_manager.cc:4584] TS 467045ecb8104c5eaaa40f4b7e2e5750 (127.0.98.1:42759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c31d2557345549cd8b91b4567d64a9bb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42759: connect: Connection refused (error 111)
I20260927 21:10:58.835808 19497 ts_tablet_manager.cc:1905] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:10:58.837705 19497 ts_tablet_manager.cc:1918] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:10:58.837790 19497 log.cc:1196] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/c31d2557345549cd8b91b4567d64a9bb
I20260927 21:10:58.838047 19497 ts_tablet_manager.cc:1939] T c31d2557345549cd8b91b4567d64a9bb P e08fe5a3d787451dbbecd1277ee9b7af: Deleting consensus metadata
I20260927 21:10:58.838497 18992 catalog_manager.cc:4841] TS e08fe5a3d787451dbbecd1277ee9b7af (127.0.98.3:43835): tablet c31d2557345549cd8b91b4567d64a9bb (table test-table [id=cd32f58461da4acda8c771e8ef703c64]) successfully deleted
I20260927 21:10:58.841199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19330
I20260927 21:10:58.845818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 18977
2026-09-27T21:10:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (754 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:10:58 dist-test-slave-8bgs krb5kdc[19501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:10:58 dist-test-slave-8bgs krb5kdc[19501](info): set up 2 sockets
Sep 27 21:10:58 dist-test-slave-8bgs krb5kdc[19501](info): commencing operation
krb5kdc: starting...
W20260927 21:10:59.883272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:10:59 dist-test-slave-8bgs krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:10:59.993978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:00.074194 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:00.074460 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:00.074497 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:00.075709 19514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:00.075785 19514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:00.075807 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:00.075829 19514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:00.075860 19514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:00.077386 19514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:00.077695 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:00.077937 19514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:00.082857 19522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.082864 19519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.082864 19520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:00.083215 19514 server_base.cc:1034] running on GCE node
I20260927 21:11:00.083485 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:00.083784 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:00.084973 19514 hybrid_clock.cc:648] HybridClock initialized: now 1790543460084955 us; error 27 us; skew 500 ppm
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:00.094185 19514 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:00.094661 19514 webserver.cc:460] Webserver started at http://127.0.98.62:41451/ using document root <none> and password file <none>
I20260927 21:11:00.094911 19514 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:00.094980 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:00.095180 19514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:00.096246 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "a586d3429bc24c50b37450650425166b"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "598871d8d159fe5167871174455367d3"
server_key_iv: "7bb4500860559098fcfe0a559f0e3e9f"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.096622 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a586d3429bc24c50b37450650425166b"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "598871d8d159fe5167871174455367d3"
server_key_iv: "7bb4500860559098fcfe0a559f0e3e9f"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.097963 19514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:00.098742 19529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.098927 19514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:00.099031 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "a586d3429bc24c50b37450650425166b"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "598871d8d159fe5167871174455367d3"
server_key_iv: "7bb4500860559098fcfe0a559f0e3e9f"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.099125 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:00.118256 19514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:00.120185 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:00.120368 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:00.124424 19514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33661
I20260927 21:11:00.124562 19581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33661 every 8 connection(s)
I20260927 21:11:00.124863 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:00.125810 19582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:00.128181 19582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b: Bootstrap starting.
I20260927 21:11:00.128770 19582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:00.128962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19514
I20260927 21:11:00.129081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:00.129105 19582 log.cc:824] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:00.129307 392 external_mini_cluster.cc:1428] Setting key 73a25bf2fb73d47b4dad3b5e6f794df9
I20260927 21:11:00.130000 19582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b: No bootstrap required, opened a new log
I20260927 21:11:00.132644 19582 raft_consensus.cc:348] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a586d3429bc24c50b37450650425166b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 21:11:00.132833 19582 raft_consensus.cc:374] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:00.132869 19582 raft_consensus.cc:729] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a586d3429bc24c50b37450650425166b, State: Initialized, Role: FOLLOWER
I20260927 21:11:00.132999 19582 consensus_queue.cc:260] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a586d3429bc24c50b37450650425166b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 21:11:00.133066 19582 raft_consensus.cc:388] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:00.133093 19582 raft_consensus.cc:482] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:00.133136 19582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:00.133795 19582 raft_consensus.cc:504] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a586d3429bc24c50b37450650425166b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 21:11:00.133886 19582 leader_election.cc:302] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a586d3429bc24c50b37450650425166b; no voters:
I20260927 21:11:00.134092 19582 leader_election.cc:288] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:00.134166 19587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:00.134334 19587 raft_consensus.cc:686] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [term 1 LEADER]: Becoming Leader. State: Replica: a586d3429bc24c50b37450650425166b, State: Running, Role: LEADER
I20260927 21:11:00.134370 19582 sys_catalog.cc:564] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:00.134418 19587 consensus_queue.cc:237] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a586d3429bc24c50b37450650425166b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 21:11:00.134771 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a586d3429bc24c50b37450650425166b. Latest consensus state: current_term: 1 leader_uuid: "a586d3429bc24c50b37450650425166b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a586d3429bc24c50b37450650425166b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } } }
I20260927 21:11:00.134935 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:00.135447 19596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:00.135918 19596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:00.137702 19596 catalog_manager.cc:1347] Generated new cluster ID: 87ed90e35ffa4d98887277e73479fc1c
I20260927 21:11:00.137770 19596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:00.137738 19589 sys_catalog.cc:455] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a586d3429bc24c50b37450650425166b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a586d3429bc24c50b37450650425166b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } } }
I20260927 21:11:00.137823 19589 sys_catalog.cc:458] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:00.140833 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.129871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.98.62:33661)
0927 21:11:00.130037 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.130046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.130081 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:11:00.130502 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.130508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.130553 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.130667 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:00.131496 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.132384 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.133438 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.133841 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.133991 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.137083 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:00.137087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:00.137087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:00.137098 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:00.139189 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.139587 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.139591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.139592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.139703 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.139910 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.139911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.139911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.140221 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:00.140387 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.140482 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.140621 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260927 21:11:00.145031 19596 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:00.145659 19596 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:00.148288 19596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b: Generated new TSK 0
I20260927 21:11:00.148473 19596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 21:11:00.177142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:00.260068 19608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:00.260286 19608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:00.260311 19608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:00.261570 19608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:00.261610 19608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:00.261637 19608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:11:00.263310 19608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:00.263569 19608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:00.263813 19608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:00.268723 19613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.268826 19614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.268744 19616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:00.268931 19608 server_base.cc:1034] running on GCE node
I20260927 21:11:00.269218 19608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:00.269488 19608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:00.270646 19608 hybrid_clock.cc:648] HybridClock initialized: now 1790543460270633 us; error 43 us; skew 500 ppm
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:00.273131 19608 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 21:11:00.273619 19608 webserver.cc:460] Webserver started at http://127.0.98.1:35821/ using document root <none> and password file <none>
I20260927 21:11:00.273910 19608 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:00.273981 19608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:00.274127 19608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:00.275243 19608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "f0984d35c47597ce393332b8e300d062"
server_key_iv: "c38a3db7c485921de4aeeca9b796432b"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.275609 19608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "f0984d35c47597ce393332b8e300d062"
server_key_iv: "c38a3db7c485921de4aeeca9b796432b"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.276916 19608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:11:00.277696 19623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.277863 19608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:00.277968 19608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "f0984d35c47597ce393332b8e300d062"
server_key_iv: "c38a3db7c485921de4aeeca9b796432b"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.278044 19608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:00.293005 19608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:00.294863 19608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:00.295045 19608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:00.295328 19608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:11:00.295639 19608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:11:00.295692 19608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.295737 19608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:11:00.295759 19608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.301810 19608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35855
I20260927 21:11:00.301887 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35855 every 8 connection(s)
I20260927 21:11:00.302263 19608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:00.311463 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.303040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54975 (local address 127.0.98.62:33661)
0927 21:11:00.303129 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.303131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.303343 (+ 212us) server_negotiation.cc:407] Connection header received
0927 21:11:00.304274 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.304275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.304286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.304338 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:00.304940 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.305462 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.306468 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.306666 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.306718 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.308877 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:00.308879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:00.308879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:00.308886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:00.310191 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.310565 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.310566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.310566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.310614 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.310847 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.310847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.310848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.310957 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:00.311006 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.311324 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.311397 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:11:00.311681 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.303046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33661 (local address 127.0.98.1:54975)
0927 21:11:00.303232 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.303270 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.304180 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.304395 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.304398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.304657 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:00.304861 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.304866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.305542 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.305543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.306388 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.306390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.306737 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.306738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.306852 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.307366 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:00.307374 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:00.308765 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:00.310265 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:00.310272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:00.310274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:00.310499 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:00.310673 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:00.310673 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:00.310674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:00.310777 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:00.311012 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:00.311017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:00.311207 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.311298 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.311426 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260927 21:11:00.312258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19608
I20260927 21:11:00.312283 19737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33661
I20260927 21:11:00.312390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:11:00.312392 19737 heartbeater.cc:464] Registering TS with master...
I20260927 21:11:00.312568 19737 heartbeater.cc:511] Master 127.0.98.62:33661 requested a full tablet report, sending...
I20260927 21:11:00.312614 392 external_mini_cluster.cc:1428] Setting key dab2671fee5fbde413191892c92afa48
I20260927 21:11:00.313115 19546 ts_manager.cc:194] Registered new tserver with Master: d8904ad10c9c4c5abe2beb47b6d05fa9 (127.0.98.1:35855)
I20260927 21:11:00.314011 19546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54975
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 21:11:00.336213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:00.416563 19742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:00.416808 19742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:00.416849 19742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:00.417997 19742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:00.418067 19742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:00.418113 19742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:11:00.419683 19742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:00.419940 19742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:00.420176 19742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:00.425117 19748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:00.425141 19742 server_base.cc:1034] running on GCE node
W20260927 21:11:00.425112 19750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.425112 19747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:00.425669 19742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:00.425884 19742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:00.427024 19742 hybrid_clock.cc:648] HybridClock initialized: now 1790543460427013 us; error 27 us; skew 500 ppm
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:00.429483 19742 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 21:11:00.429958 19742 webserver.cc:460] Webserver started at http://127.0.98.2:40793/ using document root <none> and password file <none>
I20260927 21:11:00.430208 19742 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:00.430258 19742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:00.430430 19742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:00.431557 19742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "dbf20cab78a84e15987c8ace93976c4f"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "b174dcf9f684ecff1184b3ffeb823c86"
server_key_iv: "a9b9f0d20ca38b51e41b4de281a766ff"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.431912 19742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dbf20cab78a84e15987c8ace93976c4f"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "b174dcf9f684ecff1184b3ffeb823c86"
server_key_iv: "a9b9f0d20ca38b51e41b4de281a766ff"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.433241 19742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:00.434098 19757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.434471 19742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:11:00.434576 19742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "dbf20cab78a84e15987c8ace93976c4f"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "b174dcf9f684ecff1184b3ffeb823c86"
server_key_iv: "a9b9f0d20ca38b51e41b4de281a766ff"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.434667 19742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:00.459121 19742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:00.461078 19742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:00.461251 19742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:00.461508 19742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:11:00.461829 19742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:11:00.461884 19742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.461925 19742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:11:00.461967 19742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.468405 19742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41747
I20260927 21:11:00.468451 19870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41747 every 8 connection(s)
I20260927 21:11:00.468784 19742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:11:00.471071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19742
I20260927 21:11:00.471233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:11:00.471457 392 external_mini_cluster.cc:1428] Setting key 9b5ef6d3dcaec6d53bae99d5c1a816ac
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:00.478869 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.469504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44825 (local address 127.0.98.62:33661)
0927 21:11:00.469572 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.469574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.469792 (+ 218us) server_negotiation.cc:407] Connection header received
0927 21:11:00.470734 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.470735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.470744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.470805 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:00.471614 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.472277 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.473412 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.473638 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.473698 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.475986 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:00.475988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:00.475988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:00.475994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:00.477429 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.477882 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.477883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.477884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.477944 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.478200 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.478201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.478201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.478321 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:00.478403 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.478682 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.478768 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:11:00.479143 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.469484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33661 (local address 127.0.98.2:44825)
0927 21:11:00.469661 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.469704 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.470642 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.470869 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.470872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.471241 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:00.471523 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.471531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.472374 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.472375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.473297 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.473301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.473726 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.473727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.473891 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.474448 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:00.474458 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:00.475865 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:00.477532 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:00.477540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:00.477543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:00.477782 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:00.478012 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:00.478012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:00.478013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:00.478132 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:00.478394 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:00.478400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:00.478549 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.478659 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.478846 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260927 21:11:00.479621 19871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33661
I20260927 21:11:00.479707 19871 heartbeater.cc:464] Registering TS with master...
I20260927 21:11:00.479909 19871 heartbeater.cc:511] Master 127.0.98.62:33661 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 21:11:00.480319 19546 ts_manager.cc:194] Registered new tserver with Master: dbf20cab78a84e15987c8ace93976c4f (127.0.98.2:41747)
I20260927 21:11:00.480703 19546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 21:11:00.495733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:00.577992 19876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:00.578197 19876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:00.578222 19876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:00.579329 19876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:00.579367 19876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:00.579393 19876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:11:00.580888 19876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:00.581192 19876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:00.581445 19876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:11:00.586314 19876 server_base.cc:1034] running on GCE node
W20260927 21:11:00.586280 19881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.586287 19882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:00.586290 19884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:00.586822 19876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:00.587067 19876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:00.588260 19876 hybrid_clock.cc:648] HybridClock initialized: now 1790543460588234 us; error 44 us; skew 500 ppm
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:00.590826 19876 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 21:11:00.591368 19876 webserver.cc:460] Webserver started at http://127.0.98.3:38849/ using document root <none> and password file <none>
I20260927 21:11:00.591596 19876 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:00.591660 19876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:00.591799 19876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:00.592859 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "be26bce5328144f9b4b780fa49e9fdb1"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "a7ddac25094e2342706da211e8f9b8af"
server_key_iv: "3d6066d60f6b7f9579ae60407e8674ff"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.593230 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be26bce5328144f9b4b780fa49e9fdb1"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "a7ddac25094e2342706da211e8f9b8af"
server_key_iv: "3d6066d60f6b7f9579ae60407e8674ff"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.594550 19876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:11:00.595366 19891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.595541 19876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:11:00.595639 19876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "be26bce5328144f9b4b780fa49e9fdb1"
format_stamp: "Formatted at 2026-09-27 21:11:00 on dist-test-slave-8bgs"
server_key: "a7ddac25094e2342706da211e8f9b8af"
server_key_iv: "3d6066d60f6b7f9579ae60407e8674ff"
server_key_version: "encryptionkey@0"
I20260927 21:11:00.595717 19876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:00.609855 19876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:00.612215 19876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:00.612378 19876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:00.612625 19876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:11:00.612939 19876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:11:00.612991 19876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.613034 19876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:11:00.613061 19876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:00.618732 19876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44295
I20260927 21:11:00.618785 20004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44295 every 8 connection(s)
I20260927 21:11:00.619141 19876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:11:00.620370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 19876
I20260927 21:11:00.620468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:11:00.620680 392 external_mini_cluster.cc:1428] Setting key 8df7860f236409685a47883bc2d39285
Sep 27 21:11:00 dist-test-slave-8bgs krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:00.628427 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.619860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41641 (local address 127.0.98.62:33661)
0927 21:11:00.619936 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.619938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.620113 (+ 175us) server_negotiation.cc:407] Connection header received
0927 21:11:00.621314 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.621315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.621327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.621383 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:00.622119 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.622632 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.623668 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.623868 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.623931 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.625862 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:00.625864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:00.625864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:00.625870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:00.627371 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.627697 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.627698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.627698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.627746 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.627949 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.627949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.627949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.628044 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:00.628089 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.628307 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.628366 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:11:00.628638 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.619817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33661 (local address 127.0.98.3:41641)
0927 21:11:00.619989 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.620036 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.621213 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.621472 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.621477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.621844 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:00.622041 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.622045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.622711 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.622712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.623593 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.623596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.623954 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.623955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.624048 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.624477 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:00.624485 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:00.625786 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:00.627436 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:00.627441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:00.627443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:00.627637 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:00.627800 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:00.627801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:00.627801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:00.627891 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:00.628095 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:00.628099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:00.628205 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.628284 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.628391 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260927 21:11:00.629115 20005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33661
I20260927 21:11:00.629216 20005 heartbeater.cc:464] Registering TS with master...
I20260927 21:11:00.629383 20005 heartbeater.cc:511] Master 127.0.98.62:33661 requested a full tablet report, sending...
I20260927 21:11:00.629798 19546 ts_manager.cc:194] Registered new tserver with Master: be26bce5328144f9b4b780fa49e9fdb1 (127.0.98.3:44295)
I20260927 21:11:00.630157 19546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41641
I20260927 21:11:00.632397 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:11:00.639964 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.633657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.0.98.62:33661)
0927 21:11:00.633724 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.633726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.633834 (+ 108us) server_negotiation.cc:407] Connection header received
0927 21:11:00.633927 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.633928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.633938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.634009 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:00.634688 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.635227 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.636059 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.636283 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.636386 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.637585 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:00.637587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:00.637587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:00.637595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:00.638757 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.639144 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.639145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.639146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.639204 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:00.639407 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:00.639408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:00.639408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:00.639555 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:00.639657 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.639771 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.639865 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:11:00.641371 19546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:11:00.642809 19546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:11:00.649055 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.644883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41747 (local address 127.0.0.1:43898)
0927 21:11:00.645167 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.645175 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.645261 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.645747 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.645748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.645749 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:00.645797 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.645799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.646652 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.646653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.647888 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.647891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.648741 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.648742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.648820 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.648830 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.648922 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.648973 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":52,"threads_started":1}
I20260927 21:11:00.649063 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.645350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.98.2:41747)
0927 21:11:00.645599 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.645603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.645613 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:11:00.645654 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.645656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.645672 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.645749 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:00.645862 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.646546 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.647978 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.648544 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.648643 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.648715 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.648944 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.648986 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":121,"threads_started":1}
I20260927 21:11:00.649268 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.644884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44295 (local address 127.0.0.1:36692)
0927 21:11:00.645358 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.645366 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.645436 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.645691 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.645695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.645697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:00.645764 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.645766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.647149 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.647150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.648081 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.648083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.649027 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.649028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.649087 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.649091 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.649138 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.649187 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260927 21:11:00.649447 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.644983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.0.98.1:35855)
0927 21:11:00.645172 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.645177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.645492 (+ 315us) server_negotiation.cc:407] Connection header received
0927 21:11:00.645534 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.645537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.645554 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.645620 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:00.645848 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.646954 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.648246 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.648938 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.648995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.649041 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.649309 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.649364 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260927 21:11:00.649564 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.645104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.0.98.3:44295)
0927 21:11:00.645281 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.645285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.645441 (+ 156us) server_negotiation.cc:407] Connection header received
0927 21:11:00.645496 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.645499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.645516 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.645592 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:00.645847 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.646941 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.648166 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.648865 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.648936 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.648992 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.649429 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.649472 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
I20260927 21:11:00.649269 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.644859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35855 (local address 127.0.0.1:35710)
0927 21:11:00.645125 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.645169 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.645261 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.645699 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.645700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.645701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:00.645764 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.645767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.647055 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.647056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.648162 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.648165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.649068 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.649069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.649134 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.649138 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.649181 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.649209 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":25,"thread_start_us":69,"threads_started":1}
I20260927 21:11:00.650068 19805 tablet_service.cc:1467] Processing CreateTablet for tablet 574555d279e84cfdb806a2c797bdac00 (DEFAULT_TABLE table=test-table [id=9e6c0090473c46eeb5a7bdc5e65193ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:11:00.650243 19671 tablet_service.cc:1467] Processing CreateTablet for tablet 574555d279e84cfdb806a2c797bdac00 (DEFAULT_TABLE table=test-table [id=9e6c0090473c46eeb5a7bdc5e65193ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:11:00.650347 19805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574555d279e84cfdb806a2c797bdac00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:00.650480 19671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574555d279e84cfdb806a2c797bdac00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:00.651768 19939 tablet_service.cc:1467] Processing CreateTablet for tablet 574555d279e84cfdb806a2c797bdac00 (DEFAULT_TABLE table=test-table [id=9e6c0090473c46eeb5a7bdc5e65193ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:11:00.652097 19939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574555d279e84cfdb806a2c797bdac00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:00.653229 20019 tablet_bootstrap.cc:492] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: Bootstrap starting.
I20260927 21:11:00.653784 20020 tablet_bootstrap.cc:492] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Bootstrap starting.
I20260927 21:11:00.653930 20019 tablet_bootstrap.cc:654] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:00.654500 20020 tablet_bootstrap.cc:654] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:00.654517 20019 log.cc:824] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:00.655055 20020 log.cc:824] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:00.655326 20019 tablet_bootstrap.cc:492] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: No bootstrap required, opened a new log
I20260927 21:11:00.655397 20019 ts_tablet_manager.cc:1397] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:11:00.655793 20020 tablet_bootstrap.cc:492] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: No bootstrap required, opened a new log
I20260927 21:11:00.655864 20020 ts_tablet_manager.cc:1397] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:11:00.657370 20019 raft_consensus.cc:348] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.657553 20019 raft_consensus.cc:374] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:00.657589 20019 raft_consensus.cc:729] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8904ad10c9c4c5abe2beb47b6d05fa9, State: Initialized, Role: FOLLOWER
I20260927 21:11:00.657722 20019 consensus_queue.cc:260] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.657843 20020 raft_consensus.cc:348] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.657995 20019 ts_tablet_manager.cc:1428] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:11:00.658010 20020 raft_consensus.cc:374] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:00.658046 20020 raft_consensus.cc:729] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf20cab78a84e15987c8ace93976c4f, State: Initialized, Role: FOLLOWER
I20260927 21:11:00.658013 19737 heartbeater.cc:503] Master 127.0.98.62:33661 was elected leader, sending a full tablet report...
I20260927 21:11:00.658157 20020 consensus_queue.cc:260] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.658329 20022 tablet_bootstrap.cc:492] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Bootstrap starting.
I20260927 21:11:00.658432 19871 heartbeater.cc:503] Master 127.0.98.62:33661 was elected leader, sending a full tablet report...
I20260927 21:11:00.658429 20020 ts_tablet_manager.cc:1428] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:11:00.659860 20022 tablet_bootstrap.cc:654] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:00.660358 20022 log.cc:824] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:00.661058 20022 tablet_bootstrap.cc:492] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: No bootstrap required, opened a new log
I20260927 21:11:00.661155 20022 ts_tablet_manager.cc:1397] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 21:11:00.662540 20022 raft_consensus.cc:348] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.662667 20022 raft_consensus.cc:374] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:00.662690 20022 raft_consensus.cc:729] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be26bce5328144f9b4b780fa49e9fdb1, State: Initialized, Role: FOLLOWER
I20260927 21:11:00.662823 20022 consensus_queue.cc:260] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.663074 20022 ts_tablet_manager.cc:1428] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:11:00.663146 20005 heartbeater.cc:503] Master 127.0.98.62:33661 was elected leader, sending a full tablet report...
W20260927 21:11:00.719302 19872 tablet.cc:2369] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:11:00.803011 19738 tablet.cc:2369] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:11:00.856391 20024 raft_consensus.cc:482] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:11:00.856515 20024 raft_consensus.cc:504] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.856813 20024 leader_election.cc:288] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbf20cab78a84e15987c8ace93976c4f (127.0.98.2:41747), be26bce5328144f9b4b780fa49e9fdb1 (127.0.98.3:44295)
I20260927 21:11:00.859799 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.857169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46469 (local address 127.0.98.2:41747)
0927 21:11:00.857238 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.857240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.857250 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:11:00.857295 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.857296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.857309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.857380 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:00.857547 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.858029 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.859029 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.859446 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.859538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.859548 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.859688 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.859728 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:11:00.859822 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.857039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41747 (local address 127.0.98.1:46469)
0927 21:11:00.857164 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.857176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.857234 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.857399 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.857400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.857402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:00.857478 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.857479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.858115 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.858116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.858925 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.858926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.859540 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.859541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.859616 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.859619 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.859683 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.859729 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260927 21:11:00.859820 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.857134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59397 (local address 127.0.98.3:44295)
0927 21:11:00.857195 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.857197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.857328 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:11:00.857409 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.857411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.857420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.857493 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:00.857636 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.858104 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.859058 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.859474 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.859556 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.859565 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.859718 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.859753 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:11:00.859848 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.857039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44295 (local address 127.0.98.1:59397)
0927 21:11:00.857249 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.857258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:00.857344 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:00.857497 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:00.857497 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:00.857499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:00.857568 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.857570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.858189 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.858190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.858967 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:00.858968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.859557 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.859558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:00.859643 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.859647 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:11:00.859713 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:11:00.859751 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260927 21:11:00.860173 19825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574555d279e84cfdb806a2c797bdac00" candidate_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf20cab78a84e15987c8ace93976c4f" is_pre_election: true
I20260927 21:11:00.860329 19825 raft_consensus.cc:2454] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8904ad10c9c4c5abe2beb47b6d05fa9 in term 0.
I20260927 21:11:00.860294 19959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574555d279e84cfdb806a2c797bdac00" candidate_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26bce5328144f9b4b780fa49e9fdb1" is_pre_election: true
I20260927 21:11:00.860499 19959 raft_consensus.cc:2454] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8904ad10c9c4c5abe2beb47b6d05fa9 in term 0.
I20260927 21:11:00.860610 19626 leader_election.cc:302] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8904ad10c9c4c5abe2beb47b6d05fa9, dbf20cab78a84e15987c8ace93976c4f; no voters:
I20260927 21:11:00.860769 20024 raft_consensus.cc:2781] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:11:00.860831 20024 raft_consensus.cc:482] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:11:00.860867 20024 raft_consensus.cc:3037] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:00.861497 20024 raft_consensus.cc:504] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.861644 20024 leader_election.cc:288] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [CANDIDATE]: Term 1 election: Requested vote from peers dbf20cab78a84e15987c8ace93976c4f (127.0.98.2:41747), be26bce5328144f9b4b780fa49e9fdb1 (127.0.98.3:44295)
I20260927 21:11:00.861790 19825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574555d279e84cfdb806a2c797bdac00" candidate_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf20cab78a84e15987c8ace93976c4f"
I20260927 21:11:00.861809 19959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574555d279e84cfdb806a2c797bdac00" candidate_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26bce5328144f9b4b780fa49e9fdb1"
I20260927 21:11:00.861899 19825 raft_consensus.cc:3037] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:00.861897 19959 raft_consensus.cc:3037] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:00.862617 19825 raft_consensus.cc:2454] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8904ad10c9c4c5abe2beb47b6d05fa9 in term 1.
I20260927 21:11:00.862620 19959 raft_consensus.cc:2454] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8904ad10c9c4c5abe2beb47b6d05fa9 in term 1.
I20260927 21:11:00.862829 19626 leader_election.cc:302] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8904ad10c9c4c5abe2beb47b6d05fa9, dbf20cab78a84e15987c8ace93976c4f; no voters:
I20260927 21:11:00.862982 20024 raft_consensus.cc:2781] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:00.863217 20024 raft_consensus.cc:686] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 1 LEADER]: Becoming Leader. State: Replica: d8904ad10c9c4c5abe2beb47b6d05fa9, State: Running, Role: LEADER
I20260927 21:11:00.863327 20024 consensus_queue.cc:237] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } }
I20260927 21:11:00.864194 19545 catalog_manager.cc:5495] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8904ad10c9c4c5abe2beb47b6d05fa9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8904ad10c9c4c5abe2beb47b6d05fa9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } health_report { overall_health: UNKNOWN } } }
W20260927 21:11:00.869609 20006 tablet.cc:2369] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:11:00.874166 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.872133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.0.98.1:35855)
0927 21:11:00.872182 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:00.872183 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:00.872192 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:11:00.872217 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:00.872218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:00.872233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:00.872283 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:00.872384 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.872810 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.873519 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:00.873700 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:00.873753 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:00.873884 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:00.873971 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:00.874028 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:00.874060 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 21:11:00.874075 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:11:00.875921 19959 raft_consensus.cc:1261] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 1 FOLLOWER]: Refusing update from remote peer d8904ad10c9c4c5abe2beb47b6d05fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:00.875922 19825 raft_consensus.cc:1261] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 1 FOLLOWER]: Refusing update from remote peer d8904ad10c9c4c5abe2beb47b6d05fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:00.876240 20024 consensus_queue.cc:1035] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be26bce5328144f9b4b780fa49e9fdb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:00.876394 20030 consensus_queue.cc:1035] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbf20cab78a84e15987c8ace93976c4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:00.878852 20034 mvcc.cc:205] Tried to move back new op lower bound from 7334066015745073152 to 7334066015696814080. Current Snapshot: MvccSnapshot[applied={T|T < 7334066015745073152}]
I20260927 21:11:00.881147 19545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:11:00.881232 19545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:11:00.881963 19545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a586d3429bc24c50b37450650425166b: Sending DeleteTablet for 3 replicas of tablet 574555d279e84cfdb806a2c797bdac00
I20260927 21:11:00.882246 19671 tablet_service.cc:1514] Processing DeleteTablet for tablet 574555d279e84cfdb806a2c797bdac00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:11:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35710
I20260927 21:11:00.882349 19805 tablet_service.cc:1514] Processing DeleteTablet for tablet 574555d279e84cfdb806a2c797bdac00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:11:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43898
I20260927 21:11:00.882473 19939 tablet_service.cc:1514] Processing DeleteTablet for tablet 574555d279e84cfdb806a2c797bdac00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:11:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36692
I20260927 21:11:00.882581 20043 tablet_replica.cc:330] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9: stopping tablet replica
I20260927 21:11:00.882670 20043 raft_consensus.cc:2229] T 574555d279e84cfdb806a2c797bdac00 P d8904ad10c9c4c5abe2beb47b6d05fa9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:11:00.882673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19608
I20260927 21:11:00.883422 20045 tablet_replica.cc:330] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: stopping tablet replica
I20260927 21:11:00.883519 20045 raft_consensus.cc:2229] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:11:00.883610 20045 raft_consensus.cc:2258] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:11:00.883764 20044 tablet_replica.cc:330] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: stopping tablet replica
I20260927 21:11:00.883862 20044 raft_consensus.cc:2229] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:11:00.883966 20044 raft_consensus.cc:2258] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:11:00.884063 20045 ts_tablet_manager.cc:1905] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:11:00.884370 20044 ts_tablet_manager.cc:1905] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:11:00.885753 20045 ts_tablet_manager.cc:1918] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:11:00.885828 20045 log.cc:1196] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/574555d279e84cfdb806a2c797bdac00
I20260927 21:11:00.885986 20044 ts_tablet_manager.cc:1918] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:11:00.886059 20044 log.cc:1196] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/574555d279e84cfdb806a2c797bdac00
I20260927 21:11:00.886094 20045 ts_tablet_manager.cc:1939] T 574555d279e84cfdb806a2c797bdac00 P be26bce5328144f9b4b780fa49e9fdb1: Deleting consensus metadata
I20260927 21:11:00.886358 20044 ts_tablet_manager.cc:1939] T 574555d279e84cfdb806a2c797bdac00 P dbf20cab78a84e15987c8ace93976c4f: Deleting consensus metadata
I20260927 21:11:00.886541 19531 catalog_manager.cc:4841] TS be26bce5328144f9b4b780fa49e9fdb1 (127.0.98.3:44295): tablet 574555d279e84cfdb806a2c797bdac00 (table test-table [id=9e6c0090473c46eeb5a7bdc5e65193ff]) successfully deleted
I20260927 21:11:00.886724 19532 catalog_manager.cc:4841] TS dbf20cab78a84e15987c8ace93976c4f (127.0.98.2:41747): tablet 574555d279e84cfdb806a2c797bdac00 (table test-table [id=9e6c0090473c46eeb5a7bdc5e65193ff]) successfully deleted
W20260927 21:11:00.889066 19530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35855 (error 108)
I20260927 21:11:00.889232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19742
I20260927 21:11:00.889950 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:00.889794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35855 (local address 127.0.0.1:35730)
0927 21:11:00.889857 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:00.889898 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:11:00.890079 19530 catalog_manager.cc:4584] TS d8904ad10c9c4c5abe2beb47b6d05fa9 (127.0.98.1:35855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 574555d279e84cfdb806a2c797bdac00: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35855: connect: Connection refused (error 111)
I20260927 21:11:00.894531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19876
I20260927 21:11:00.899186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 19514
2026-09-27T21:11:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T21:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:00Z Disabled control of system clock
I20260927 21:11:00.926287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:11:01.005846 20050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:01.006057 20050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:01.006078 20050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:01.007197 20050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:01.007234 20050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:01.007247 20050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:01.007261 20050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:01.007272 20050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:01.008517 20050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:01.008689 20050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:01.008910 20050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:01.013571 20056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:01.013624 20058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:01.013703 20050 server_base.cc:1034] running on GCE node
W20260927 21:11:01.013571 20055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:01.014134 20050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:01.014483 20050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:01.015691 20050 hybrid_clock.cc:648] HybridClock initialized: now 1790543461015672 us; error 34 us; skew 500 ppm
I20260927 21:11:01.016819 20050 webserver.cc:460] Webserver started at http://127.0.98.62:43787/ using document root <none> and password file <none>
I20260927 21:11:01.017060 20050 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:01.017107 20050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:01.017246 20050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:01.018247 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "203e8052675cf9f7e4cdd66f2f877c48"
server_key_iv: "fa7f5b989ee1f71c00a42368518ada29"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.018592 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "203e8052675cf9f7e4cdd66f2f877c48"
server_key_iv: "fa7f5b989ee1f71c00a42368518ada29"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.019955 20050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:11:01.020670 20064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.020848 20050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:01.020941 20050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "203e8052675cf9f7e4cdd66f2f877c48"
server_key_iv: "fa7f5b989ee1f71c00a42368518ada29"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.021013 20050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:01.032680 20050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:01.032989 20050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:01.033144 20050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:01.037025 20050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38129
I20260927 21:11:01.037055 20116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38129 every 8 connection(s)
I20260927 21:11:01.037407 20050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:01.038324 20117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:01.040421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20050
I20260927 21:11:01.040508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:01.040730 392 external_mini_cluster.cc:1428] Setting key 0a14aa784d76d3ddcee7fc4505ad5662
I20260927 21:11:01.040830 20117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c: Bootstrap starting.
I20260927 21:11:01.041584 20117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:01.041899 20117 log.cc:824] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:01.042639 20117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c: No bootstrap required, opened a new log
I20260927 21:11:01.044898 20117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260927 21:11:01.045064 20117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:01.045097 20117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbac109fdc4bb5b9bbf88c63f1b74c, State: Initialized, Role: FOLLOWER
I20260927 21:11:01.045234 20117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260927 21:11:01.045310 20117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:01.045338 20117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:01.045395 20117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:01.046120 20117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260927 21:11:01.046248 20117 leader_election.cc:302] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30fbac109fdc4bb5b9bbf88c63f1b74c; no voters:
I20260927 21:11:01.046427 20117 leader_election.cc:288] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:01.046497 20122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:01.046644 20122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [term 1 LEADER]: Becoming Leader. State: Replica: 30fbac109fdc4bb5b9bbf88c63f1b74c, State: Running, Role: LEADER
I20260927 21:11:01.046712 20117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:01.046768 20122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260927 21:11:01.047116 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.041276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.98.62:38129)
0927 21:11:01.041517 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.041525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.041565 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:11:01.042163 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.042170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.042230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.042393 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:01.042591 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.044000 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.044762 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.045389 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.045512 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.046188 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:01.046192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:01.046192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:01.046405 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:01.046448 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:01.046463 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:01.046532 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.046790 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.046882 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260927 21:11:01.047358 20123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } } }
I20260927 21:11:01.047470 20123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:01.047369 20124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30fbac109fdc4bb5b9bbf88c63f1b74c. Latest consensus state: current_term: 1 leader_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fbac109fdc4bb5b9bbf88c63f1b74c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } } }
I20260927 21:11:01.047551 20124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:01.047799 20131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:01.048331 20131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:01.049782 20131 catalog_manager.cc:1347] Generated new cluster ID: 2b4b2d9294c741bd898c82ed9e88fd45
I20260927 21:11:01.049832 20131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:01.057049 20131 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:01.057698 20131 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:01.062225 20131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c: Generated new TSK 0
I20260927 21:11:01.062395 20131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:01.065701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--builtin_ntp_servers=127.0.98.20:45105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:11:01.159955 20141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:01.160149 20141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:01.160171 20141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:01.161337 20141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:01.161376 20141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:01.161402 20141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 21:11:01.162738 20141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:01.162973 20141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:01.163306 20141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:01.168354 20146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:01.168349 20149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:01.168356 20147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:01.168812 20141 server_base.cc:1034] running on GCE node
I20260927 21:11:01.169034 20141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:01.169275 20141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:01.170444 20141 hybrid_clock.cc:648] HybridClock initialized: now 1790543461170430 us; error 31 us; skew 500 ppm
I20260927 21:11:01.171813 20141 webserver.cc:460] Webserver started at http://127.0.98.1:33115/ using document root <none> and password file <none>
I20260927 21:11:01.172060 20141 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:01.172132 20141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:01.172286 20141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:01.173357 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data/instance:
uuid: "dc2dcdeee2d649dd9c02925f5568a17f"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "11a429f6e4a937ba2dafa4fd7526132b"
server_key_iv: "06264bacac2da199958dfd8733f82df9"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.173720 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dc2dcdeee2d649dd9c02925f5568a17f"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "11a429f6e4a937ba2dafa4fd7526132b"
server_key_iv: "06264bacac2da199958dfd8733f82df9"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.175026 20141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:11:01.175866 20155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.176102 20141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:11:01.176198 20141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal
uuid: "dc2dcdeee2d649dd9c02925f5568a17f"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "11a429f6e4a937ba2dafa4fd7526132b"
server_key_iv: "06264bacac2da199958dfd8733f82df9"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.176266 20141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:01.196141 20141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:01.196444 20141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:01.196556 20141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:01.196758 20141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:11:01.197126 20141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:11:01.197163 20141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.197186 20141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:11:01.197245 20141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.203069 20141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45159
I20260927 21:11:01.203159 20268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45159 every 8 connection(s)
I20260927 21:11:01.203529 20141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 21:11:01.204162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20141
I20260927 21:11:01.204272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-0/wal/instance
I20260927 21:11:01.204510 392 external_mini_cluster.cc:1428] Setting key 3b8e03dcce831d9007858ed75f0c3901
I20260927 21:11:01.205499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--builtin_ntp_servers=127.0.98.20:45105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:11:01.209229 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.204330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44115 (local address 127.0.98.62:38129)
0927 21:11:01.204426 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.204428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.204701 (+ 273us) server_negotiation.cc:407] Connection header received
0927 21:11:01.205366 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.205367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.205379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.205415 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:01.206187 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.206699 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.207798 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.208003 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.208080 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.208935 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:01.208936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:01.208937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:01.208941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:01.208955 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:01.208959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:01.209027 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.209122 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.209151 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:11:01.209470 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.204330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38129 (local address 127.0.98.1:44115)
0927 21:11:01.204531 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.204592 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.205262 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.205501 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.205505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.205521 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:01.206094 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.206099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.206794 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.206796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.207707 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.207709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.208087 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.208088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.208225 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.208777 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:11:01.208777 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:01.208811 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:11:01.208833 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:01.209027 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:01.209034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:01.209038 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.209126 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.209226 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260927 21:11:01.210008 20269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38129
I20260927 21:11:01.210103 20269 heartbeater.cc:464] Registering TS with master...
I20260927 21:11:01.210273 20269 heartbeater.cc:511] Master 127.0.98.62:38129 requested a full tablet report, sending...
I20260927 21:11:01.210863 20081 ts_manager.cc:194] Registered new tserver with Master: dc2dcdeee2d649dd9c02925f5568a17f (127.0.98.1:45159)
I20260927 21:11:01.211892 20081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44115
W20260927 21:11:01.286422 20272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:01.286631 20272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:01.286653 20272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:01.287731 20272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:01.287766 20272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:01.287797 20272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 21:11:01.289047 20272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:01.289228 20272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:01.289470 20272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:01.294358 20280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:01.294421 20277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:01.294564 20272 server_base.cc:1034] running on GCE node
W20260927 21:11:01.294370 20278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:01.295416 20272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:01.295677 20272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:01.296839 20272 hybrid_clock.cc:648] HybridClock initialized: now 1790543461296817 us; error 35 us; skew 500 ppm
I20260927 21:11:01.298178 20272 webserver.cc:460] Webserver started at http://127.0.98.2:38369/ using document root <none> and password file <none>
I20260927 21:11:01.298441 20272 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:01.298514 20272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:01.298659 20272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:01.299791 20272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data/instance:
uuid: "1547a7e0d9e54319bbef8db8e1e03d8d"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "e90aa4699419401c905c1ba69a658f69"
server_key_iv: "15ec57a44cedc825949e088936f21031"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.300160 20272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1547a7e0d9e54319bbef8db8e1e03d8d"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "e90aa4699419401c905c1ba69a658f69"
server_key_iv: "15ec57a44cedc825949e088936f21031"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.301497 20272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:01.302214 20286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.302398 20272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:01.302496 20272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal
uuid: "1547a7e0d9e54319bbef8db8e1e03d8d"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "e90aa4699419401c905c1ba69a658f69"
server_key_iv: "15ec57a44cedc825949e088936f21031"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.302574 20272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:01.327934 20272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:01.328267 20272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:01.328435 20272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:01.328675 20272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:11:01.329015 20272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:11:01.329066 20272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.329100 20272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:11:01.329128 20272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.334671 20272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44239
I20260927 21:11:01.334759 20399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44239 every 8 connection(s)
I20260927 21:11:01.335182 20272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 21:11:01.340032 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.335957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54487 (local address 127.0.98.62:38129)
0927 21:11:01.336026 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.336029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.336249 (+ 220us) server_negotiation.cc:407] Connection header received
0927 21:11:01.336712 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.336713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.336722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.336782 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:01.337419 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.337895 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.338823 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.339000 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.339050 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.339807 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:01.339808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:01.339808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:01.339812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:01.339826 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:01.339829 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:01.339872 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.339949 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.339974 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:11:01.340248 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.335928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38129 (local address 127.0.98.2:54487)
0927 21:11:01.336118 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.336159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.336630 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.336837 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.336841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.336848 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:01.337340 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.337345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.337976 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.337976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.338743 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.338745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.339073 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.339074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.339196 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.339682 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:11:01.339682 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:01.339710 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:11:01.339731 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:01.339882 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:01.339885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:01.339888 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.339939 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.340022 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260927 21:11:01.340418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20272
I20260927 21:11:01.340515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal/instance
I20260927 21:11:01.340731 20400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38129
I20260927 21:11:01.340754 392 external_mini_cluster.cc:1428] Setting key c3208e43be336a36ba76318cb04fa543
I20260927 21:11:01.340817 20400 heartbeater.cc:464] Registering TS with master...
I20260927 21:11:01.341024 20400 heartbeater.cc:511] Master 127.0.98.62:38129 requested a full tablet report, sending...
I20260927 21:11:01.341426 20081 ts_manager.cc:194] Registered new tserver with Master: 1547a7e0d9e54319bbef8db8e1e03d8d (127.0.98.2:44239)
I20260927 21:11:01.341709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--builtin_ntp_servers=127.0.98.20:45105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:11:01.341743 20081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54487
W20260927 21:11:01.421284 20403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:01.421494 20403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:01.421515 20403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:01.422603 20403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:01.422645 20403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:01.422669 20403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 21:11:01.424001 20403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:01.424271 20403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:01.424541 20403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:01.429425 20409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:01.429440 20411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:01.429435 20408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:01.429661 20403 server_base.cc:1034] running on GCE node
I20260927 21:11:01.429844 20403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:01.430034 20403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:01.431151 20403 hybrid_clock.cc:648] HybridClock initialized: now 1790543461431129 us; error 34 us; skew 500 ppm
I20260927 21:11:01.432397 20403 webserver.cc:460] Webserver started at http://127.0.98.3:38021/ using document root <none> and password file <none>
I20260927 21:11:01.432641 20403 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:01.432699 20403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:01.432915 20403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:01.433962 20403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data/instance:
uuid: "d30ac8f698294095b1122c6e2382c774"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "c9ea4ce7e7844255520dc3dba38e691e"
server_key_iv: "7f9c688dd89427abc792bb48672b82ea"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.434311 20403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d30ac8f698294095b1122c6e2382c774"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "c9ea4ce7e7844255520dc3dba38e691e"
server_key_iv: "7f9c688dd89427abc792bb48672b82ea"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.435673 20403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:11:01.436427 20417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.436661 20403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:11:01.436789 20403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal
uuid: "d30ac8f698294095b1122c6e2382c774"
format_stamp: "Formatted at 2026-09-27 21:11:01 on dist-test-slave-8bgs"
server_key: "c9ea4ce7e7844255520dc3dba38e691e"
server_key_iv: "7f9c688dd89427abc792bb48672b82ea"
server_key_version: "encryptionkey@0"
I20260927 21:11:01.436864 20403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:01.446455 20403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:01.446907 20403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:01.447077 20403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:01.447403 20403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:11:01.447798 20403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:11:01.447860 20403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.447901 20403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:11:01.447924 20403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:01.454391 20403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46433
I20260927 21:11:01.454437 20530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46433 every 8 connection(s)
I20260927 21:11:01.454808 20403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 21:11:01.456238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20403
I20260927 21:11:01.456365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal/instance
I20260927 21:11:01.456600 392 external_mini_cluster.cc:1428] Setting key e3c066cdcdae687f7827e9f189a44334
I20260927 21:11:01.460503 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.455599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41457 (local address 127.0.98.62:38129)
0927 21:11:01.455672 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.455675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.455939 (+ 264us) server_negotiation.cc:407] Connection header received
0927 21:11:01.456518 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.456520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.456534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.456613 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:01.457539 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.458120 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.459074 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.459364 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.459429 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.460259 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:01.460260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:01.460261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:01.460265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:01.460279 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:01.460282 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:01.460330 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.460424 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.460445 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:11:01.460700 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.455605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38129 (local address 127.0.98.3:41457)
0927 21:11:01.455809 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.455845 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.456408 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.456738 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.456742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.456765 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:01.457424 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.457430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.458196 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.458197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.458979 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.458980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.459444 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.459444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.459556 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.460124 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:11:01.460125 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:01.460154 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:11:01.460177 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:01.460338 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:01.460342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:01.460344 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.460395 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.460473 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260927 21:11:01.461162 20531 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38129
I20260927 21:11:01.461256 20531 heartbeater.cc:464] Registering TS with master...
I20260927 21:11:01.461436 20531 heartbeater.cc:511] Master 127.0.98.62:38129 requested a full tablet report, sending...
I20260927 21:11:01.461831 20081 ts_manager.cc:194] Registered new tserver with Master: d30ac8f698294095b1122c6e2382c774 (127.0.98.3:46433)
I20260927 21:11:01.462208 20081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41457
I20260927 21:11:01.468472 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:11:01.472483 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.469805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.0.98.62:38129)
0927 21:11:01.469876 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.469878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.469938 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:11:01.469975 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.469976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.469984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.470039 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:01.470176 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.470642 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.471370 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.471552 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.471605 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.472268 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:01.472269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:01.472269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:01.472273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:01.472282 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:01.472285 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:01.472320 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.472366 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.472384 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:11:01.473996 20081 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:11:01.475539 20081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:11:01.481927 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.478175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.98.3:46433)
0927 21:11:01.478389 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.478394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.478403 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:11:01.478439 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.478441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.478456 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.478665 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:01.478721 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.479486 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.481006 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.481565 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.481624 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.481687 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.481831 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.481858 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260927 21:11:01.481953 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.477979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46433 (local address 127.0.0.1:49778)
0927 21:11:01.478104 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.478115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.478177 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.478562 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.478563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.478564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:01.478593 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.478594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.479592 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.479593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.480920 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.480923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.481648 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.481649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.481731 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.481743 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.481821 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.481875 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260927 21:11:01.482749 20465 tablet_service.cc:1467] Processing CreateTablet for tablet 08faa4c8f46c45e9810811ff6f1b4fb3 (DEFAULT_TABLE table=test-table [id=6a24ebf448924077b67f3166dac53750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:11:01.482899 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.477747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45159 (local address 127.0.0.1:60520)
0927 21:11:01.478056 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.478101 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.478174 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.478486 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.478489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.478492 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:01.478575 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.478580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.479933 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.479933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.481016 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.481018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.482646 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.482647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.482711 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.482716 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.482767 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.482809 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260927 21:11:01.483072 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.477860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.2:44239)
0927 21:11:01.478016 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.478021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.478176 (+ 155us) server_negotiation.cc:407] Connection header received
0927 21:11:01.478302 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.478304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.478321 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.478378 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:01.478681 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.479803 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.481074 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.481656 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.481727 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.481818 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.482931 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.482980 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260927 21:11:01.483165 20465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08faa4c8f46c45e9810811ff6f1b4fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:01.483211 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.477845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.98.1:45159)
0927 21:11:01.478057 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.478061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.478257 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:11:01.478303 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.478305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.478321 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.478407 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:01.478648 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.479852 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.481853 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.482567 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.482642 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.482720 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.483061 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.483127 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":103,"threads_started":1}
I20260927 21:11:01.483289 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.477763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44239 (local address 127.0.0.1:44954)
0927 21:11:01.478103 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.478115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.478202 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.478492 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.478493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.478495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:01.478575 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.478577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.479904 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.479905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.480972 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.480973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.482760 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.482761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.482824 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.482828 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.483174 (+ 346us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.483221 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260927 21:11:01.485724 20545 tablet_bootstrap.cc:492] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Bootstrap starting.
I20260927 21:11:01.486373 20545 tablet_bootstrap.cc:654] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:01.486320 20334 tablet_service.cc:1467] Processing CreateTablet for tablet 08faa4c8f46c45e9810811ff6f1b4fb3 (DEFAULT_TABLE table=test-table [id=6a24ebf448924077b67f3166dac53750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:11:01.486505 20203 tablet_service.cc:1467] Processing CreateTablet for tablet 08faa4c8f46c45e9810811ff6f1b4fb3 (DEFAULT_TABLE table=test-table [id=6a24ebf448924077b67f3166dac53750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:11:01.486699 20334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08faa4c8f46c45e9810811ff6f1b4fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:01.486732 20545 log.cc:824] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:01.486800 20203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08faa4c8f46c45e9810811ff6f1b4fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:01.487419 20545 tablet_bootstrap.cc:492] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: No bootstrap required, opened a new log
I20260927 21:11:01.487504 20545 ts_tablet_manager.cc:1397] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:11:01.489308 20545 raft_consensus.cc:348] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.489461 20545 raft_consensus.cc:374] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:01.489506 20545 raft_consensus.cc:729] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d30ac8f698294095b1122c6e2382c774, State: Initialized, Role: FOLLOWER
I20260927 21:11:01.489639 20545 consensus_queue.cc:260] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.489933 20545 ts_tablet_manager.cc:1428] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:11:01.490072 20531 heartbeater.cc:503] Master 127.0.98.62:38129 was elected leader, sending a full tablet report...
I20260927 21:11:01.490474 20547 tablet_bootstrap.cc:492] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Bootstrap starting.
I20260927 21:11:01.491151 20547 tablet_bootstrap.cc:654] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:01.491245 20548 tablet_bootstrap.cc:492] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: Bootstrap starting.
I20260927 21:11:01.491469 20547 log.cc:824] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:01.491816 20548 tablet_bootstrap.cc:654] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:01.492117 20548 log.cc:824] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:01.492136 20547 tablet_bootstrap.cc:492] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: No bootstrap required, opened a new log
I20260927 21:11:01.492194 20547 ts_tablet_manager.cc:1397] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:11:01.492780 20548 tablet_bootstrap.cc:492] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: No bootstrap required, opened a new log
I20260927 21:11:01.492859 20548 ts_tablet_manager.cc:1397] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:11:01.494119 20547 raft_consensus.cc:348] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.494429 20547 raft_consensus.cc:374] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:01.494495 20547 raft_consensus.cc:729] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1547a7e0d9e54319bbef8db8e1e03d8d, State: Initialized, Role: FOLLOWER
I20260927 21:11:01.494740 20547 consensus_queue.cc:260] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.495044 20547 ts_tablet_manager.cc:1428] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:11:01.495054 20548 raft_consensus.cc:348] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.495218 20400 heartbeater.cc:503] Master 127.0.98.62:38129 was elected leader, sending a full tablet report...
I20260927 21:11:01.495245 20548 raft_consensus.cc:374] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:01.495298 20548 raft_consensus.cc:729] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2dcdeee2d649dd9c02925f5568a17f, State: Initialized, Role: FOLLOWER
I20260927 21:11:01.495417 20548 consensus_queue.cc:260] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.495674 20548 ts_tablet_manager.cc:1428] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:11:01.498456 20269 heartbeater.cc:503] Master 127.0.98.62:38129 was elected leader, sending a full tablet report...
W20260927 21:11:01.585765 20401 tablet.cc:2369] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:11:01.695237 20553 raft_consensus.cc:482] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:11:01.695389 20553 raft_consensus.cc:504] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.695793 20553 leader_election.cc:288] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1547a7e0d9e54319bbef8db8e1e03d8d (127.0.98.2:44239), d30ac8f698294095b1122c6e2382c774 (127.0.98.3:46433)
I20260927 21:11:01.698823 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.695976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46433 (local address 127.0.98.1:41265)
0927 21:11:01.696043 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.696057 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.696110 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.696298 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.696299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.696301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:01.696386 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.696389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.697049 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.697051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.697836 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.697838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.698581 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.698582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.698655 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.698660 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.698714 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.698768 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":22}
I20260927 21:11:01.698848 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.696029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41265 (local address 127.0.98.3:46433)
0927 21:11:01.696071 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.696073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.696092 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:11:01.696199 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.696200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.696210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.696279 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:01.696464 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.696973 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.698050 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.698489 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.698571 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.698581 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.698733 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.698773 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:01.699024 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.696050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34427 (local address 127.0.98.2:44239)
0927 21:11:01.696100 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.696102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.696284 (+ 182us) server_negotiation.cc:407] Connection header received
0927 21:11:01.696321 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.696322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.696331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.696397 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:11:01.696564 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.697041 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.698059 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.698633 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.698706 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.698716 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.698893 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.698937 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:11:01.699043 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.695982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44239 (local address 127.0.98.1:34427)
0927 21:11:01.696205 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.696216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:01.696274 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:01.696398 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:01.696399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:01.696405 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:11:01.696495 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.696496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.697131 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.697132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.697845 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:01.697847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.698731 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.698733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:01.698809 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.698813 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:11:01.698870 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:11:01.698921 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":98,"thread_start_us":70,"threads_started":1}
I20260927 21:11:01.699235 20485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08faa4c8f46c45e9810811ff6f1b4fb3" candidate_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d30ac8f698294095b1122c6e2382c774" is_pre_election: true
I20260927 21:11:01.699384 20485 raft_consensus.cc:2454] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc2dcdeee2d649dd9c02925f5568a17f in term 0.
I20260927 21:11:01.699630 20159 leader_election.cc:302] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d30ac8f698294095b1122c6e2382c774, dc2dcdeee2d649dd9c02925f5568a17f; no voters:
I20260927 21:11:01.699751 20553 raft_consensus.cc:2781] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:11:01.699807 20553 raft_consensus.cc:482] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:11:01.699746 20354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08faa4c8f46c45e9810811ff6f1b4fb3" candidate_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" is_pre_election: true
I20260927 21:11:01.699833 20553 raft_consensus.cc:3037] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:01.699903 20354 raft_consensus.cc:2454] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc2dcdeee2d649dd9c02925f5568a17f in term 0.
I20260927 21:11:01.700484 20553 raft_consensus.cc:504] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.700611 20553 leader_election.cc:288] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [CANDIDATE]: Term 1 election: Requested vote from peers 1547a7e0d9e54319bbef8db8e1e03d8d (127.0.98.2:44239), d30ac8f698294095b1122c6e2382c774 (127.0.98.3:46433)
I20260927 21:11:01.700721 20354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08faa4c8f46c45e9810811ff6f1b4fb3" candidate_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d"
I20260927 21:11:01.700735 20485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08faa4c8f46c45e9810811ff6f1b4fb3" candidate_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d30ac8f698294095b1122c6e2382c774"
I20260927 21:11:01.700799 20354 raft_consensus.cc:3037] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:01.700830 20485 raft_consensus.cc:3037] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:01.701500 20354 raft_consensus.cc:2454] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc2dcdeee2d649dd9c02925f5568a17f in term 1.
I20260927 21:11:01.701624 20485 raft_consensus.cc:2454] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc2dcdeee2d649dd9c02925f5568a17f in term 1.
I20260927 21:11:01.701691 20157 leader_election.cc:302] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1547a7e0d9e54319bbef8db8e1e03d8d, dc2dcdeee2d649dd9c02925f5568a17f; no voters:
I20260927 21:11:01.701797 20553 raft_consensus.cc:2781] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:01.701922 20553 raft_consensus.cc:686] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 1 LEADER]: Becoming Leader. State: Replica: dc2dcdeee2d649dd9c02925f5568a17f, State: Running, Role: LEADER
I20260927 21:11:01.702018 20553 consensus_queue.cc:237] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } }
I20260927 21:11:01.702841 20081 catalog_manager.cc:5495] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f reported cstate change: term changed from 0 to 1, leader changed from <none> to dc2dcdeee2d649dd9c02925f5568a17f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2dcdeee2d649dd9c02925f5568a17f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 } health_report { overall_health: UNKNOWN } } }
W20260927 21:11:01.704182 20270 tablet.cc:2369] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:11:01.705369 20532 tablet.cc:2369] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:11:01.708089 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.705932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.98.1:45159)
0927 21:11:01.705981 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:01.705983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:01.705991 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:11:01.706021 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:01.706021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:01.706045 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:01.706107 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:01.706211 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.706722 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.707482 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:01.707654 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:01.707705 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:01.707834 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:01.707921 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:01.707980 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:01.708013 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 21:11:01.708029 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:11:01.709671 20485 raft_consensus.cc:1261] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 1 FOLLOWER]: Refusing update from remote peer dc2dcdeee2d649dd9c02925f5568a17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:01.709686 20354 raft_consensus.cc:1261] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 1 FOLLOWER]: Refusing update from remote peer dc2dcdeee2d649dd9c02925f5568a17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:01.709966 20553 consensus_queue.cc:1035] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d30ac8f698294095b1122c6e2382c774" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:01.710083 20555 consensus_queue.cc:1035] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1547a7e0d9e54319bbef8db8e1e03d8d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:01.712380 20558 mvcc.cc:205] Tried to move back new op lower bound from 7334066019160518656 to 7334066019132182528. Current Snapshot: MvccSnapshot[applied={T|T < 7334066019160518656}]
I20260927 21:11:01.712946 20559 mvcc.cc:205] Tried to move back new op lower bound from 7334066019160518656 to 7334066019132182528. Current Snapshot: MvccSnapshot[applied={T|T < 7334066019160518656}]
I20260927 21:11:01.714498 20081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:11:01.714586 20081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:11:01.715355 20081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30fbac109fdc4bb5b9bbf88c63f1b74c: Sending DeleteTablet for 3 replicas of tablet 08faa4c8f46c45e9810811ff6f1b4fb3
I20260927 21:11:01.715667 20203 tablet_service.cc:1514] Processing DeleteTablet for tablet 08faa4c8f46c45e9810811ff6f1b4fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:11:01 UTC) from {username='slave'} at 127.0.0.1:60520
I20260927 21:11:01.715696 20334 tablet_service.cc:1514] Processing DeleteTablet for tablet 08faa4c8f46c45e9810811ff6f1b4fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:11:01 UTC) from {username='slave'} at 127.0.0.1:44954
I20260927 21:11:01.715900 20569 tablet_replica.cc:330] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: stopping tablet replica
I20260927 21:11:01.715876 20465 tablet_service.cc:1514] Processing DeleteTablet for tablet 08faa4c8f46c45e9810811ff6f1b4fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:11:01 UTC) from {username='slave'} at 127.0.0.1:49778
I20260927 21:11:01.716051 20569 raft_consensus.cc:2229] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:11:01.716091 20570 tablet_replica.cc:330] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: stopping tablet replica
I20260927 21:11:01.716219 20568 tablet_replica.cc:330] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f: stopping tablet replica
I20260927 21:11:01.716252 20569 raft_consensus.cc:2258] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:11:01.716311 20568 raft_consensus.cc:2229] T 08faa4c8f46c45e9810811ff6f1b4fb3 P dc2dcdeee2d649dd9c02925f5568a17f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:11:01.716387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20141
I20260927 21:11:01.717440 20569 ts_tablet_manager.cc:1905] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:11:01.719004 20569 ts_tablet_manager.cc:1918] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:11:01.719076 20569 log.cc:1196] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-1/wal/wals/08faa4c8f46c45e9810811ff6f1b4fb3
I20260927 21:11:01.719334 20569 ts_tablet_manager.cc:1939] T 08faa4c8f46c45e9810811ff6f1b4fb3 P 1547a7e0d9e54319bbef8db8e1e03d8d: Deleting consensus metadata
I20260927 21:11:01.719748 20066 catalog_manager.cc:4841] TS 1547a7e0d9e54319bbef8db8e1e03d8d (127.0.98.2:44239): tablet 08faa4c8f46c45e9810811ff6f1b4fb3 (table test-table [id=6a24ebf448924077b67f3166dac53750]) successfully deleted
W20260927 21:11:01.721395 20068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45159 (error 108)
I20260927 21:11:01.721907 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:01.721793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45159 (local address 127.0.0.1:60544)
0927 21:11:01.721848 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:01.721882 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:11:01.722028 20068 catalog_manager.cc:4584] TS dc2dcdeee2d649dd9c02925f5568a17f (127.0.98.1:45159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08faa4c8f46c45e9810811ff6f1b4fb3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45159: connect: Connection refused (error 111)
I20260927 21:11:01.722385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20272
I20260927 21:11:01.722398 20570 raft_consensus.cc:2229] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:11:01.722528 20570 raft_consensus.cc:2258] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:11:01.723603 20570 ts_tablet_manager.cc:1905] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:11:01.724901 20570 ts_tablet_manager.cc:1918] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:11:01.724985 20570 log.cc:1196] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Deleting WAL directory at /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790543220945705-392-0/minicluster-data/ts-2/wal/wals/08faa4c8f46c45e9810811ff6f1b4fb3
I20260927 21:11:01.725229 20570 ts_tablet_manager.cc:1939] T 08faa4c8f46c45e9810811ff6f1b4fb3 P d30ac8f698294095b1122c6e2382c774: Deleting consensus metadata
I20260927 21:11:01.725632 20068 catalog_manager.cc:4841] TS d30ac8f698294095b1122c6e2382c774 (127.0.98.3:46433): tablet 08faa4c8f46c45e9810811ff6f1b4fb3 (table test-table [id=6a24ebf448924077b67f3166dac53750]) successfully deleted
I20260927 21:11:01.727825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20403
I20260927 21:11:01.732311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20050
2026-09-27T21:11:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (831 ms)
[----------] 6 tests from EncryptionPolicyTest (8302 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:01 dist-test-slave-8bgs krb5kdc[20574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:01 dist-test-slave-8bgs krb5kdc[20574](info): set up 2 sockets
Sep 27 21:11:01 dist-test-slave-8bgs krb5kdc[20574](info): commencing operation
krb5kdc: starting...
W20260927 21:11:02.759539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:02 dist-test-slave-8bgs krb5kdc[20574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:02.873323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:02.952445 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:02.952694 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:02.952734 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:02.953897 20587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:02.953970 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:02.953996 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:02.954028 20587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:02.954063 20587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:02.955621 20587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:02.955883 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:02.956112 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:02.960762 20595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:02.960788 20587 server_base.cc:1034] running on GCE node
W20260927 21:11:02.960954 20592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:02.960853 20593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:02.961295 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:02.961611 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:02.962778 20587 hybrid_clock.cc:648] HybridClock initialized: now 1790543462962754 us; error 35 us; skew 500 ppm
Sep 27 21:11:02 dist-test-slave-8bgs krb5kdc[20574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:02.965183 20587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:02.965622 20587 webserver.cc:460] Webserver started at http://127.0.0.1:34273/ using document root <none> and password file <none>
I20260927 21:11:02.965854 20587 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:02.965921 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:02.966070 20587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:02.967135 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "d0d64052322e4ef39d60e6b6e2b4b903"
format_stamp: "Formatted at 2026-09-27 21:11:02 on dist-test-slave-8bgs"
server_key: "6e774aee2f5df1d7719c3511fbb77a70"
server_key_iv: "14e655fa04dee1e5be41e351063fe74f"
server_key_version: "encryptionkey@0"
I20260927 21:11:02.967522 20587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0d64052322e4ef39d60e6b6e2b4b903"
format_stamp: "Formatted at 2026-09-27 21:11:02 on dist-test-slave-8bgs"
server_key: "6e774aee2f5df1d7719c3511fbb77a70"
server_key_iv: "14e655fa04dee1e5be41e351063fe74f"
server_key_version: "encryptionkey@0"
I20260927 21:11:02.968930 20587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:11:02.969615 20602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:02.969791 20587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:02.969887 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "d0d64052322e4ef39d60e6b6e2b4b903"
format_stamp: "Formatted at 2026-09-27 21:11:02 on dist-test-slave-8bgs"
server_key: "6e774aee2f5df1d7719c3511fbb77a70"
server_key_iv: "14e655fa04dee1e5be41e351063fe74f"
server_key_version: "encryptionkey@0"
I20260927 21:11:02.969972 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:02.986855 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:02.989264 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:02.989426 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:02.993275 20587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41921
I20260927 21:11:02.993328 20654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41921 every 8 connection(s)
I20260927 21:11:02.993692 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:02.994565 20655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:02.996851 20655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903: Bootstrap starting.
I20260927 21:11:02.997414 20655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:02.997701 20655 log.cc:824] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:02.997812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20587
I20260927 21:11:02.997956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:02.998157 392 external_mini_cluster.cc:1428] Setting key 445d60c40577dbfd5bb61f3bd19d505a
I20260927 21:11:02.998452 20655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903: No bootstrap required, opened a new log
I20260927 21:11:03.000414 20655 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260927 21:11:03.000571 20655 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:03.000619 20655 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d64052322e4ef39d60e6b6e2b4b903, State: Initialized, Role: FOLLOWER
I20260927 21:11:03.000752 20655 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260927 21:11:03.000815 20655 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:03.000836 20655 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:03.000900 20655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:03.001454 20655 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260927 21:11:03.001538 20655 leader_election.cc:302] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d64052322e4ef39d60e6b6e2b4b903; no voters:
I20260927 21:11:03.001708 20655 leader_election.cc:288] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:03.001741 20660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:03.001914 20660 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [term 1 LEADER]: Becoming Leader. State: Replica: d0d64052322e4ef39d60e6b6e2b4b903, State: Running, Role: LEADER
I20260927 21:11:03.001965 20655 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:03.002017 20660 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } }
I20260927 21:11:03.002364 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d64052322e4ef39d60e6b6e2b4b903. Latest consensus state: current_term: 1 leader_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } } }
I20260927 21:11:03.002637 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:03.002554 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d64052322e4ef39d60e6b6e2b4b903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41921 } } }
I20260927 21:11:03.002892 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:03.003046 20669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:03.003618 20669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:03.005285 20669 catalog_manager.cc:1347] Generated new cluster ID: 1f52f30de6324f64b9b911fd8983ecc9
I20260927 21:11:03.005336 20669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:11:03 dist-test-slave-8bgs krb5kdc[20574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:11:03.009958 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:02.998662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47749 (local address 127.0.0.1:41921)
0927 21:11:02.998819 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:02.998829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:02.998865 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:11:02.999753 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:02.999761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:02.999810 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:02.999955 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:03.000656 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:03.002183 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:03.003309 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:03.003958 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:03.004089 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:03.006767 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:03.006770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:03.006771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:03.006782 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:03.008453 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:03.008848 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:03.008851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:03.008852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:03.008936 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:03.009172 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:03.009173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:03.009173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:03.009399 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:03.009585 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:03.009692 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:11:03.009796 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260927 21:11:03.010896 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:03.021085 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:03.012233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41889 (local address 127.0.0.1:41921)
0927 21:11:03.012271 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:03.012273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:03.015250 (+ 2977us) server_negotiation.cc:407] Connection header received
0927 21:11:03.015319 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:03.015320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:03.015330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:03.015406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:03.015910 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:03.016394 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:03.017083 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:03.017305 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:03.017379 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:03.018521 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:03.018522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:03.018522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:03.018529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:03.019951 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:03.020307 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:03.020308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:03.020309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:03.020360 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:03.020583 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:03.020584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:03.020585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:03.020692 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:03.020798 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:03.020919 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:11:03.020979 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:11:03.045686 20669 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:03.046270 20669 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:03.052119 20669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d64052322e4ef39d60e6b6e2b4b903: Generated new TSK 0
I20260927 21:11:03.052321 20669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:03.055325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20587
W20260927 21:11:03.059438 20572 connection.cc:535] client connection to 127.0.0.1:41921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41921): Connection reset by peer (error 104)
2026-09-27T21:11:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:03 dist-test-slave-8bgs krb5kdc[20686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:03 dist-test-slave-8bgs krb5kdc[20686](info): set up 2 sockets
Sep 27 21:11:03 dist-test-slave-8bgs krb5kdc[20686](info): commencing operation
krb5kdc: starting...
W20260927 21:11:04.091053 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:04 dist-test-slave-8bgs krb5kdc[20686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:04.202299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:04.280983 20699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:04.281219 20699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:04.281258 20699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:04.282497 20699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:04.282572 20699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:04.282598 20699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:04.282626 20699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:04.282650 20699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:04.284176 20699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:04.284454 20699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:04.284684 20699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:04.289749 20707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:04.289726 20704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:04.289719 20705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:04.289814 20699 server_base.cc:1034] running on GCE node
I20260927 21:11:04.290213 20699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:04.290496 20699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:04.291714 20699 hybrid_clock.cc:648] HybridClock initialized: now 1790543464291677 us; error 35 us; skew 500 ppm
Sep 27 21:11:04 dist-test-slave-8bgs krb5kdc[20686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:04.294091 20699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:04.294557 20699 webserver.cc:460] Webserver started at http://127.0.0.1:46825/ using document root <none> and password file <none>
I20260927 21:11:04.294781 20699 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:04.294848 20699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:04.295001 20699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:04.296079 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "4ada8ed5611d483dbdb81ba87e40c720"
format_stamp: "Formatted at 2026-09-27 21:11:04 on dist-test-slave-8bgs"
server_key: "6b819f13cb758e0c6f6d5a78f2b0c8e4"
server_key_iv: "800eeb5206b704d3d0d5f25fb3ea2a97"
server_key_version: "encryptionkey@0"
I20260927 21:11:04.296442 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ada8ed5611d483dbdb81ba87e40c720"
format_stamp: "Formatted at 2026-09-27 21:11:04 on dist-test-slave-8bgs"
server_key: "6b819f13cb758e0c6f6d5a78f2b0c8e4"
server_key_iv: "800eeb5206b704d3d0d5f25fb3ea2a97"
server_key_version: "encryptionkey@0"
I20260927 21:11:04.297709 20699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:04.298434 20714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:04.298625 20699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:04.298725 20699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "4ada8ed5611d483dbdb81ba87e40c720"
format_stamp: "Formatted at 2026-09-27 21:11:04 on dist-test-slave-8bgs"
server_key: "6b819f13cb758e0c6f6d5a78f2b0c8e4"
server_key_iv: "800eeb5206b704d3d0d5f25fb3ea2a97"
server_key_version: "encryptionkey@0"
I20260927 21:11:04.298812 20699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:04.332343 20699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:04.334473 20699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:04.334650 20699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:04.338842 20699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34227
I20260927 21:11:04.338881 20766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34227 every 8 connection(s)
I20260927 21:11:04.339295 20699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:04.340137 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:04.342361 20767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720: Bootstrap starting.
I20260927 21:11:04.342932 20767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:04.343314 20767 log.cc:824] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:04.343961 20767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720: No bootstrap required, opened a new log
I20260927 21:11:04.345553 20767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ada8ed5611d483dbdb81ba87e40c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260927 21:11:04.345690 20767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:04.345713 20767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ada8ed5611d483dbdb81ba87e40c720, State: Initialized, Role: FOLLOWER
I20260927 21:11:04.345876 20767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ada8ed5611d483dbdb81ba87e40c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260927 21:11:04.345937 20767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:04.345956 20767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:04.346010 20767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:04.346529 20767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ada8ed5611d483dbdb81ba87e40c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260927 21:11:04.346638 20767 leader_election.cc:302] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ada8ed5611d483dbdb81ba87e40c720; no voters:
I20260927 21:11:04.346844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20699
I20260927 21:11:04.346851 20767 leader_election.cc:288] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:04.346882 20770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:04.346969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:04.346998 20770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [term 1 LEADER]: Becoming Leader. State: Replica: 4ada8ed5611d483dbdb81ba87e40c720, State: Running, Role: LEADER
I20260927 21:11:04.347070 20770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ada8ed5611d483dbdb81ba87e40c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } }
I20260927 21:11:04.347158 20767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:04.347203 392 external_mini_cluster.cc:1428] Setting key 41abb539e15fa42645477052d89ae2ce
I20260927 21:11:04.347384 20770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ada8ed5611d483dbdb81ba87e40c720. Latest consensus state: current_term: 1 leader_uuid: "4ada8ed5611d483dbdb81ba87e40c720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ada8ed5611d483dbdb81ba87e40c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } } }
I20260927 21:11:04.347530 20770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:04.347798 20778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:04.348184 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ada8ed5611d483dbdb81ba87e40c720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ada8ed5611d483dbdb81ba87e40c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34227 } } }
I20260927 21:11:04.348313 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:04.348354 20778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:04.349920 20778 catalog_manager.cc:1347] Generated new cluster ID: 5b241505be6344d2baf9f052cd6779e0
I20260927 21:11:04.349973 20778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:04.353276 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:04.347856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37203 (local address 127.0.0.1:34227)
0927 21:11:04.348459 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:04.348468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:04.348806 (+ 338us) server_negotiation.cc:407] Connection header received
0927 21:11:04.348974 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:04.348981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:04.349042 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:04.349190 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:04.349446 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:04.350530 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:04.351424 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:04.351851 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:04.351967 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:04.352664 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:04.352668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:04.352668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:04.352838 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:04.352867 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:04.352877 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:04.352970 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:04.353047 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 21:11:04.353077 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":89,"threads_started":1}
I20260927 21:11:04.354184 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:04.361874 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:04.355482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54421 (local address 127.0.0.1:34227)
0927 21:11:04.355520 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:04.355522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:04.359258 (+ 3736us) server_negotiation.cc:407] Connection header received
0927 21:11:04.359330 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:04.359331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:04.359353 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:04.359401 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:04.359496 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:04.359971 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:04.360686 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:04.360904 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:04.360978 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:04.361634 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:04.361636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:04.361636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:04.361640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:04.361653 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:04.361657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:04.361709 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:04.361768 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 21:11:04.361780 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:11:04.374429 20778 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:04.374962 20778 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:04.378579 20778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ada8ed5611d483dbdb81ba87e40c720: Generated new TSK 0
I20260927 21:11:04.378791 20778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:04.381681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20699
W20260927 21:11:04.385512 20684 connection.cc:535] client connection to 127.0.0.1:34227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34227): Connection reset by peer (error 104)
2026-09-27T21:11:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:04 dist-test-slave-8bgs krb5kdc[20798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:04 dist-test-slave-8bgs krb5kdc[20798](info): set up 2 sockets
Sep 27 21:11:04 dist-test-slave-8bgs krb5kdc[20798](info): commencing operation
krb5kdc: starting...
W20260927 21:11:05.403265 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:05 dist-test-slave-8bgs krb5kdc[20798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:05.523239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:05.606503 20811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:05.606716 20811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:05.606738 20811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:05.607838 20811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:05.607874 20811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:05.607888 20811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:05.607901 20811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:05.607913 20811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:05.609272 20811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:05.609450 20811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:05.609617 20811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:05.614208 20817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:05.614229 20816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:05.614274 20819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:05.614334 20811 server_base.cc:1034] running on GCE node
I20260927 21:11:05.614650 20811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:05.614961 20811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:05.616129 20811 hybrid_clock.cc:648] HybridClock initialized: now 1790543465616118 us; error 39 us; skew 500 ppm
Sep 27 21:11:05 dist-test-slave-8bgs krb5kdc[20798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:05.618455 20811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:05.618883 20811 webserver.cc:460] Webserver started at http://127.0.0.1:42189/ using document root <none> and password file <none>
I20260927 21:11:05.619158 20811 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:05.619227 20811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:05.619383 20811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:05.620386 20811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "860607b47c6945cebedea0c4814704e0"
format_stamp: "Formatted at 2026-09-27 21:11:05 on dist-test-slave-8bgs"
server_key: "4bf6ab1f9ebfd52950cf83c287f92d91"
server_key_iv: "e474ca58d68424e7994dba28b48f05f2"
server_key_version: "encryptionkey@0"
I20260927 21:11:05.620754 20811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "860607b47c6945cebedea0c4814704e0"
format_stamp: "Formatted at 2026-09-27 21:11:05 on dist-test-slave-8bgs"
server_key: "4bf6ab1f9ebfd52950cf83c287f92d91"
server_key_iv: "e474ca58d68424e7994dba28b48f05f2"
server_key_version: "encryptionkey@0"
I20260927 21:11:05.622015 20811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:11:05.622681 20826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:05.622855 20811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:05.622954 20811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "860607b47c6945cebedea0c4814704e0"
format_stamp: "Formatted at 2026-09-27 21:11:05 on dist-test-slave-8bgs"
server_key: "4bf6ab1f9ebfd52950cf83c287f92d91"
server_key_iv: "e474ca58d68424e7994dba28b48f05f2"
server_key_version: "encryptionkey@0"
I20260927 21:11:05.623034 20811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:05.626192 20811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:05.628017 20811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:05.628167 20811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:05.631994 20811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38509
I20260927 21:11:05.632027 20878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38509 every 8 connection(s)
I20260927 21:11:05.632328 20811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:05.633284 20879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:05.635402 20879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0: Bootstrap starting.
I20260927 21:11:05.635913 20879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:05.636142 20879 log.cc:824] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:05.636730 20879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0: No bootstrap required, opened a new log
I20260927 21:11:05.637759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20811
I20260927 21:11:05.637897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:05.638098 392 external_mini_cluster.cc:1428] Setting key 61dc8135b495ff037ae5a9e8add307bb
I20260927 21:11:05.638751 20879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860607b47c6945cebedea0c4814704e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260927 21:11:05.638890 20879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:05.638912 20879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860607b47c6945cebedea0c4814704e0, State: Initialized, Role: FOLLOWER
I20260927 21:11:05.639042 20879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860607b47c6945cebedea0c4814704e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260927 21:11:05.639150 20879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:05.639246 20879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:05.639309 20879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:05.640280 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:05.638636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57015 (local address 127.0.0.1:38509)
0927 21:11:05.638819 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:05.638826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:05.638850 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:11:05.638980 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:05.638986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:05.639026 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:05.639211 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:05.639816 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:05.639820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:05.639820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:05.639824 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:05.639884 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:05.639895 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:05.639964 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:05.640059 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 21:11:05.640099 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260927 21:11:05.640388 20879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860607b47c6945cebedea0c4814704e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260927 21:11:05.640494 20879 leader_election.cc:302] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 860607b47c6945cebedea0c4814704e0; no voters:
I20260927 21:11:05.640676 20879 leader_election.cc:288] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:05.640730 20884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:05.640865 20884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [term 1 LEADER]: Becoming Leader. State: Replica: 860607b47c6945cebedea0c4814704e0, State: Running, Role: LEADER
I20260927 21:11:05.640910 20879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:05.640980 20884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860607b47c6945cebedea0c4814704e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260927 21:11:05.641289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:05.641360 20884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860607b47c6945cebedea0c4814704e0. Latest consensus state: current_term: 1 leader_uuid: "860607b47c6945cebedea0c4814704e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860607b47c6945cebedea0c4814704e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } } }
I20260927 21:11:05.641373 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "860607b47c6945cebedea0c4814704e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860607b47c6945cebedea0c4814704e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } } }
I20260927 21:11:05.641450 20884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:05.641456 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0 [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:05.642823 20899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:05.642921 20899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:05.643041 20891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:05.643220 20891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:05.644932 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:05.642928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49795 (local address 127.0.0.1:38509)
0927 21:11:05.642962 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:05.642963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:05.643951 (+ 988us) server_negotiation.cc:407] Connection header received
0927 21:11:05.644032 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:05.644033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:05.644038 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:05.644110 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:05.644674 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:05.644676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:05.644676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:05.644680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:05.644693 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:05.644698 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:05.644749 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:05.644855 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:11:05.644868 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:05.645031 20891 catalog_manager.cc:1347] Generated new cluster ID: 398d58ec9ef84f84bce4621dcbfd820a
I20260927 21:11:05.645071 20891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:05.649374 20891 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:05.649542 20891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:05.655162 20891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 860607b47c6945cebedea0c4814704e0: Generated new TSK 0
I20260927 21:11:05.655335 20891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:05.657492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20811
2026-09-27T21:11:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:05 dist-test-slave-8bgs krb5kdc[20910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:05 dist-test-slave-8bgs krb5kdc[20910](info): set up 2 sockets
Sep 27 21:11:05 dist-test-slave-8bgs krb5kdc[20910](info): commencing operation
krb5kdc: starting...
W20260927 21:11:06.767148 392 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:06 dist-test-slave-8bgs krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:06.882397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:06.960922 20923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:06.961156 20923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:06.961195 20923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:06.962320 20923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:06.962387 20923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:06.962414 20923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:06.962446 20923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:06.962471 20923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:06.963951 20923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:06.964226 20923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:06.964447 20923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:06.969070 20929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:06.969059 20931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:06.969060 20928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:06.969321 20923 server_base.cc:1034] running on GCE node
I20260927 21:11:06.969581 20923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:06.969892 20923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:06.971055 20923 hybrid_clock.cc:648] HybridClock initialized: now 1790543466971032 us; error 35 us; skew 500 ppm
Sep 27 21:11:06 dist-test-slave-8bgs krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:06.973417 20923 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:06.973825 20923 webserver.cc:460] Webserver started at http://127.0.0.1:33399/ using document root <none> and password file <none>
I20260927 21:11:06.974088 20923 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:06.974156 20923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:06.974310 20923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:06.975435 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "5e09a0a323074eff843011217a8065f3"
format_stamp: "Formatted at 2026-09-27 21:11:06 on dist-test-slave-8bgs"
server_key: "532e6ea977fcc42368ed63d299155134"
server_key_iv: "6e037ef61a67e906feb598bc0130f995"
server_key_version: "encryptionkey@0"
I20260927 21:11:06.975786 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e09a0a323074eff843011217a8065f3"
format_stamp: "Formatted at 2026-09-27 21:11:06 on dist-test-slave-8bgs"
server_key: "532e6ea977fcc42368ed63d299155134"
server_key_iv: "6e037ef61a67e906feb598bc0130f995"
server_key_version: "encryptionkey@0"
I20260927 21:11:06.977185 20923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:06.977919 20938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:06.978087 20923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:06.978185 20923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "5e09a0a323074eff843011217a8065f3"
format_stamp: "Formatted at 2026-09-27 21:11:06 on dist-test-slave-8bgs"
server_key: "532e6ea977fcc42368ed63d299155134"
server_key_iv: "6e037ef61a67e906feb598bc0130f995"
server_key_version: "encryptionkey@0"
I20260927 21:11:06.978267 20923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:06.988399 20923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:06.990062 20923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:06.990213 20923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:06.994159 20923 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43753
I20260927 21:11:06.994189 20990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43753 every 8 connection(s)
I20260927 21:11:06.994508 20923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:06.995412 20991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:06.996295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 20923
I20260927 21:11:06.996433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:06.996619 392 external_mini_cluster.cc:1428] Setting key 790444835dd6ee0942c749f8b33f7b1e
I20260927 21:11:06.997951 20991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3: Bootstrap starting.
I20260927 21:11:06.998682 20991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:06.998993 20991 log.cc:824] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:06.999652 20991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3: No bootstrap required, opened a new log
I20260927 21:11:07.001449 20991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09a0a323074eff843011217a8065f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260927 21:11:07.001644 20991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:07.001698 20991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09a0a323074eff843011217a8065f3, State: Initialized, Role: FOLLOWER
I20260927 21:11:07.001829 20991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09a0a323074eff843011217a8065f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260927 21:11:07.001928 20991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:07.001971 20991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:07.002020 20991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:07.002562 20991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09a0a323074eff843011217a8065f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260927 21:11:07.002671 20991 leader_election.cc:302] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e09a0a323074eff843011217a8065f3; no voters:
I20260927 21:11:07.002822 20991 leader_election.cc:288] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:07.002887 20996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:07.003054 20996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [term 1 LEADER]: Becoming Leader. State: Replica: 5e09a0a323074eff843011217a8065f3, State: Running, Role: LEADER
I20260927 21:11:07.003075 20991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 21:11:06 dist-test-slave-8bgs krb5kdc[20910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:11:07.003192 20996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09a0a323074eff843011217a8065f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } }
I20260927 21:11:07.003499 20996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e09a0a323074eff843011217a8065f3. Latest consensus state: current_term: 1 leader_uuid: "5e09a0a323074eff843011217a8065f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09a0a323074eff843011217a8065f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } } }
I20260927 21:11:07.003589 20996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:07.003509 20997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e09a0a323074eff843011217a8065f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09a0a323074eff843011217a8065f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43753 } } }
I20260927 21:11:07.003717 20997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3 [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:07.004961 21012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:07.005020 21012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:07.005124 21003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:07.005294 21003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:07.006698 21003 catalog_manager.cc:1347] Generated new cluster ID: bb38d5d6470940f7acefa60270b8d322
I20260927 21:11:07.006745 21003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:07.007625 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:06.997120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53949 (local address 127.0.0.1:43753)
0927 21:11:06.997293 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:06.997300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:06.997324 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:11:06.997732 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:06.997738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:06.997780 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:06.997901 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:06.998695 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:06.999736 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:07.000526 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:07.000940 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:07.001058 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:07.003863 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:07.003867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:07.003868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:07.003881 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:07.005971 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:07.006380 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:07.006382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:07.006383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:07.006496 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:07.006746 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:07.006747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:07.006748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:07.006978 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:07.007136 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:07.007276 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:11:07.007417 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260927 21:11:07.008446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:07.015115 21003 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:07.015812 21003 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:07.020802 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:07.009666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34301 (local address 127.0.0.1:43753)
0927 21:11:07.009701 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:07.009703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:07.014813 (+ 5110us) server_negotiation.cc:407] Connection header received
0927 21:11:07.014927 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:07.014928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:07.014939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:07.015008 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:07.015591 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:07.016148 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:07.016848 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:07.017050 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:07.017132 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:07.018256 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:07.018257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:07.018257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:07.018264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:07.019656 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:07.020022 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:07.020023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:07.020024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:07.020080 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:07.020276 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:07.020277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:07.020278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:07.020389 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:07.020482 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:07.020592 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:11:07.020690 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":17}
I20260927 21:11:07.027206 21003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e09a0a323074eff843011217a8065f3: Generated new TSK 0
I20260927 21:11:07.027395 21003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:07.030781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 20923
2026-09-27T21:11:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:07 dist-test-slave-8bgs krb5kdc[21022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:07 dist-test-slave-8bgs krb5kdc[21022](info): set up 2 sockets
Sep 27 21:11:07 dist-test-slave-8bgs krb5kdc[21022](info): commencing operation
krb5kdc: starting...
W20260927 21:11:08.063074 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:08 dist-test-slave-8bgs krb5kdc[21022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:08.174809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:08.253192 21035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:08.253417 21035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:08.253451 21035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:08.254557 21035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:08.254623 21035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:08.254645 21035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:08.254666 21035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:08.254695 21035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:08.256121 21035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:08.256397 21035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:08.256614 21035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:08.261287 21041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:08.261444 21040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:08.261453 21043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:08.261591 21035 server_base.cc:1034] running on GCE node
I20260927 21:11:08.261842 21035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:08.262079 21035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:08.263200 21035 hybrid_clock.cc:648] HybridClock initialized: now 1790543468263180 us; error 35 us; skew 500 ppm
Sep 27 21:11:08 dist-test-slave-8bgs krb5kdc[21022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:08.265707 21035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:08.266163 21035 webserver.cc:460] Webserver started at http://127.0.0.1:37045/ using document root <none> and password file <none>
I20260927 21:11:08.266392 21035 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:08.266438 21035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:08.266568 21035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:08.267654 21035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "7923fc640d4a4d2296a49ec60cb082ba"
format_stamp: "Formatted at 2026-09-27 21:11:08 on dist-test-slave-8bgs"
server_key: "23a09b2ed9729e172427315a0b7ea189"
server_key_iv: "51496031638e56699ae3b1d0482cb447"
server_key_version: "encryptionkey@0"
I20260927 21:11:08.268003 21035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "7923fc640d4a4d2296a49ec60cb082ba"
format_stamp: "Formatted at 2026-09-27 21:11:08 on dist-test-slave-8bgs"
server_key: "23a09b2ed9729e172427315a0b7ea189"
server_key_iv: "51496031638e56699ae3b1d0482cb447"
server_key_version: "encryptionkey@0"
I20260927 21:11:08.269330 21035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:08.270076 21050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:08.270262 21035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:08.270377 21035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "7923fc640d4a4d2296a49ec60cb082ba"
format_stamp: "Formatted at 2026-09-27 21:11:08 on dist-test-slave-8bgs"
server_key: "23a09b2ed9729e172427315a0b7ea189"
server_key_iv: "51496031638e56699ae3b1d0482cb447"
server_key_version: "encryptionkey@0"
I20260927 21:11:08.270449 21035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:08.294865 21035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:08.297245 21035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:08.297415 21035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:08.301481 21035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46139
I20260927 21:11:08.301520 21102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46139 every 8 connection(s)
I20260927 21:11:08.301879 21035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:08.302716 21103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:08.304944 21103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba: Bootstrap starting.
I20260927 21:11:08.305462 21103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:08.305693 21103 log.cc:824] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:08.306339 21103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba: No bootstrap required, opened a new log
I20260927 21:11:08.307996 21103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fc640d4a4d2296a49ec60cb082ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260927 21:11:08.308135 21103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:08.308157 21103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7923fc640d4a4d2296a49ec60cb082ba, State: Initialized, Role: FOLLOWER
I20260927 21:11:08.308260 21103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fc640d4a4d2296a49ec60cb082ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260927 21:11:08.308310 21103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:08.308329 21103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:08.308357 21103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:08.308827 21103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fc640d4a4d2296a49ec60cb082ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260927 21:11:08.308911 21103 leader_election.cc:302] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7923fc640d4a4d2296a49ec60cb082ba; no voters:
I20260927 21:11:08.309079 21103 leader_election.cc:288] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:08.309136 21106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:08.309191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21035
I20260927 21:11:08.309290 21106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [term 1 LEADER]: Becoming Leader. State: Replica: 7923fc640d4a4d2296a49ec60cb082ba, State: Running, Role: LEADER
I20260927 21:11:08.309309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:08.309350 21103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:08.309432 21106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fc640d4a4d2296a49ec60cb082ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } }
I20260927 21:11:08.309513 392 external_mini_cluster.cc:1428] Setting key 098ab104f358b43d0e0d1b7021548ba3
I20260927 21:11:08.309792 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7923fc640d4a4d2296a49ec60cb082ba. Latest consensus state: current_term: 1 leader_uuid: "7923fc640d4a4d2296a49ec60cb082ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fc640d4a4d2296a49ec60cb082ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } } }
I20260927 21:11:08.309957 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:08.310191 21108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7923fc640d4a4d2296a49ec60cb082ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fc640d4a4d2296a49ec60cb082ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46139 } } }
I20260927 21:11:08.310310 21108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:08.310725 21116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:08.311259 21116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:08.312790 21116 catalog_manager.cc:1347] Generated new cluster ID: 5061a0b196ec4775ba4aad702e67b72d
I20260927 21:11:08.312842 21116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:08.315788 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:08.310213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55495 (local address 127.0.0.1:46139)
0927 21:11:08.310628 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:08.310637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:08.310676 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:11:08.310798 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:08.310805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:08.310892 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:08.311117 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:08.311360 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:08.312540 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:08.313634 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:08.314281 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:08.314397 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:08.315121 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:08.315125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:08.315125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:08.315329 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:08.315366 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:08.315380 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:08.315452 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:08.315520 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:11:08.315583 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":70,"threads_started":1}
I20260927 21:11:08.316704 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:08.321583 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:08.318274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43933 (local address 127.0.0.1:46139)
0927 21:11:08.318305 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:08.318307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:08.318316 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:11:08.318411 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:08.318412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:08.318420 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:08.318472 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:08.318612 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:08.319327 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:08.320353 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:08.320631 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:08.320684 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:08.321354 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:08.321355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:08.321356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:08.321359 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:08.321372 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:08.321376 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:08.321428 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:08.321490 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 21:11:08.321512 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:11:08.331489 21116 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:08.331993 21116 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:08.337024 21116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7923fc640d4a4d2296a49ec60cb082ba: Generated new TSK 0
I20260927 21:11:08.337232 21116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:08.342947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21035
2026-09-27T21:11:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:08 dist-test-slave-8bgs krb5kdc[21134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:08 dist-test-slave-8bgs krb5kdc[21134](info): set up 2 sockets
Sep 27 21:11:08 dist-test-slave-8bgs krb5kdc[21134](info): commencing operation
krb5kdc: starting...
W20260927 21:11:09.363003 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:09 dist-test-slave-8bgs krb5kdc[21134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:09.476146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:09.558286 21147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:09.558517 21147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:09.558553 21147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:09.559724 21147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:09.559795 21147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:09.559821 21147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:09.559958 21147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:09.560014 21147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:09.561693 21147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:09.561872 21147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:09.562034 21147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:09.567127 21152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:09.567188 21155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:09.567127 21153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:09.567266 21147 server_base.cc:1034] running on GCE node
I20260927 21:11:09.567579 21147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:09.567859 21147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:09.569033 21147 hybrid_clock.cc:648] HybridClock initialized: now 1790543469569020 us; error 50 us; skew 500 ppm
Sep 27 21:11:09 dist-test-slave-8bgs krb5kdc[21134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:09.571358 21147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:09.571806 21147 webserver.cc:460] Webserver started at http://127.0.0.1:46293/ using document root <none> and password file <none>
I20260927 21:11:09.572041 21147 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:09.572108 21147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:09.572263 21147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:09.573292 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "f7dfc65203754f43b26e2f9e256801a2"
format_stamp: "Formatted at 2026-09-27 21:11:09 on dist-test-slave-8bgs"
server_key: "5f20230fc4661a076b64635b16fcd25f"
server_key_iv: "038c202bf6117ae844d867bb3192c740"
server_key_version: "encryptionkey@0"
I20260927 21:11:09.573647 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7dfc65203754f43b26e2f9e256801a2"
format_stamp: "Formatted at 2026-09-27 21:11:09 on dist-test-slave-8bgs"
server_key: "5f20230fc4661a076b64635b16fcd25f"
server_key_iv: "038c202bf6117ae844d867bb3192c740"
server_key_version: "encryptionkey@0"
I20260927 21:11:09.574903 21147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:11:09.575656 21162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:09.575826 21147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:09.575923 21147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "f7dfc65203754f43b26e2f9e256801a2"
format_stamp: "Formatted at 2026-09-27 21:11:09 on dist-test-slave-8bgs"
server_key: "5f20230fc4661a076b64635b16fcd25f"
server_key_iv: "038c202bf6117ae844d867bb3192c740"
server_key_version: "encryptionkey@0"
I20260927 21:11:09.576005 21147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:09.579151 21147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:09.581166 21147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:09.581267 21147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:09.584861 21147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39019
I20260927 21:11:09.584905 21214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39019 every 8 connection(s)
I20260927 21:11:09.585187 21147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:09.586143 21215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:09.588335 21215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2: Bootstrap starting.
I20260927 21:11:09.588876 21215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:09.589162 21215 log.cc:824] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:09.589872 21215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2: No bootstrap required, opened a new log
I20260927 21:11:09.590260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21147
I20260927 21:11:09.590389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:09.590575 392 external_mini_cluster.cc:1428] Setting key 750a0925ee4c302d414e49713cd6f875
I20260927 21:11:09.592041 21215 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfc65203754f43b26e2f9e256801a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260927 21:11:09.592192 21215 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:09.592260 21215 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7dfc65203754f43b26e2f9e256801a2, State: Initialized, Role: FOLLOWER
I20260927 21:11:09.592444 21215 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfc65203754f43b26e2f9e256801a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260927 21:11:09.592549 21215 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:09.592573 21215 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:09.592645 21215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:09.593408 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:09.591048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50951 (local address 127.0.0.1:39019)
0927 21:11:09.591241 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:09.591248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:09.591270 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:11:09.592049 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:09.592056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:09.592085 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:09.592207 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:09.592910 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:09.592913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:09.592913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:09.592916 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:09.592962 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:09.592976 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:09.593031 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:09.593158 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:11:09.593199 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260927 21:11:09.593871 21215 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfc65203754f43b26e2f9e256801a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260927 21:11:09.594010 21215 leader_election.cc:302] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7dfc65203754f43b26e2f9e256801a2; no voters:
I20260927 21:11:09.594261 21220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:09.594388 21220 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [term 1 LEADER]: Becoming Leader. State: Replica: f7dfc65203754f43b26e2f9e256801a2, State: Running, Role: LEADER
I20260927 21:11:09.594427 21215 leader_election.cc:288] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:09.594483 21220 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfc65203754f43b26e2f9e256801a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260927 21:11:09.594686 21215 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:09.594712 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:09.594832 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7dfc65203754f43b26e2f9e256801a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfc65203754f43b26e2f9e256801a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } } }
I20260927 21:11:09.594842 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7dfc65203754f43b26e2f9e256801a2. Latest consensus state: current_term: 1 leader_uuid: "f7dfc65203754f43b26e2f9e256801a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfc65203754f43b26e2f9e256801a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } } }
I20260927 21:11:09.594980 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:09.595012 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:09.595302 21225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:09.596076 21225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:09.598021 21225 catalog_manager.cc:1347] Generated new cluster ID: ba45b7bfc27a4e40a0781ffc9fca9e93
I20260927 21:11:09.598076 21225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:09.609251 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:09.596525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34135 (local address 127.0.0.1:39019)
0927 21:11:09.596564 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:09.596566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:09.597503 (+ 937us) server_negotiation.cc:407] Connection header received
0927 21:11:09.597662 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:09.597663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:09.597668 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:09.597731 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:09.607376 (+ 9645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:09.607378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:09.607379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:09.607384 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:09.607403 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:09.607408 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:09.607459 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:09.609158 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0927 21:11:09.609175 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:11:09.620831 21225 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:09.621063 21225 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:09.634845 21225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7dfc65203754f43b26e2f9e256801a2: Generated new TSK 0
I20260927 21:11:09.635057 21225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:09.641054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21147
2026-09-27T21:11:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:09 dist-test-slave-8bgs krb5kdc[21246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:09 dist-test-slave-8bgs krb5kdc[21246](info): set up 2 sockets
Sep 27 21:11:09 dist-test-slave-8bgs krb5kdc[21246](info): commencing operation
krb5kdc: starting...
W20260927 21:11:10.659031 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:10 dist-test-slave-8bgs krb5kdc[21246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:10.769774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:10.849843 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:10.850041 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:10.850064 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:10.851241 21259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:10.851282 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:10.851296 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:10.851351 21259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:10.851378 21259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:10.852864 21259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:10.853121 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:10.853345 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:10.858215 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:10.858191 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:10.858204 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:10.858516 21259 server_base.cc:1034] running on GCE node
I20260927 21:11:10.858798 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:10.859169 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:10.860325 21259 hybrid_clock.cc:648] HybridClock initialized: now 1790543470860313 us; error 29 us; skew 500 ppm
Sep 27 21:11:10 dist-test-slave-8bgs krb5kdc[21246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:10.867225 21259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:10.867638 21259 webserver.cc:460] Webserver started at http://127.0.0.1:46575/ using document root <none> and password file <none>
I20260927 21:11:10.867868 21259 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:10.867913 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:10.868055 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:10.869102 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "0631bc0bd6984c54ae9964794df3a086"
format_stamp: "Formatted at 2026-09-27 21:11:10 on dist-test-slave-8bgs"
server_key: "587c8c91922e02f0d4a6cd79986f9a4b"
server_key_iv: "4eeed27484f956a117361a0cef410111"
server_key_version: "encryptionkey@0"
I20260927 21:11:10.869455 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "0631bc0bd6984c54ae9964794df3a086"
format_stamp: "Formatted at 2026-09-27 21:11:10 on dist-test-slave-8bgs"
server_key: "587c8c91922e02f0d4a6cd79986f9a4b"
server_key_iv: "4eeed27484f956a117361a0cef410111"
server_key_version: "encryptionkey@0"
I20260927 21:11:10.870707 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:11:10.871471 21274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:10.871628 21259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:10.871721 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "0631bc0bd6984c54ae9964794df3a086"
format_stamp: "Formatted at 2026-09-27 21:11:10 on dist-test-slave-8bgs"
server_key: "587c8c91922e02f0d4a6cd79986f9a4b"
server_key_iv: "4eeed27484f956a117361a0cef410111"
server_key_version: "encryptionkey@0"
I20260927 21:11:10.871826 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:10.890353 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:10.892210 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:10.892367 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:10.896274 21259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43231
I20260927 21:11:10.896322 21326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43231 every 8 connection(s)
I20260927 21:11:10.896682 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:10.897505 21327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:10.899752 21327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086: Bootstrap starting.
I20260927 21:11:10.900311 21327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:10.900588 21327 log.cc:824] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:10.901254 21327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086: No bootstrap required, opened a new log
I20260927 21:11:10.902825 21327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0631bc0bd6984c54ae9964794df3a086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260927 21:11:10.902958 21327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:10.902981 21327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0631bc0bd6984c54ae9964794df3a086, State: Initialized, Role: FOLLOWER
I20260927 21:11:10.903219 21327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0631bc0bd6984c54ae9964794df3a086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260927 21:11:10.903299 21327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:10.903322 21327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:10.903383 21327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:10.903899 21327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0631bc0bd6984c54ae9964794df3a086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260927 21:11:10.904009 21327 leader_election.cc:302] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0631bc0bd6984c54ae9964794df3a086; no voters:
I20260927 21:11:10.904179 21327 leader_election.cc:288] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:10.904220 21330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:10.904443 21330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [term 1 LEADER]: Becoming Leader. State: Replica: 0631bc0bd6984c54ae9964794df3a086, State: Running, Role: LEADER
I20260927 21:11:10.904448 21327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:10.904589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21259
I20260927 21:11:10.904604 21330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0631bc0bd6984c54ae9964794df3a086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260927 21:11:10.904719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:10.904943 392 external_mini_cluster.cc:1428] Setting key 7256a6bbb80428dafe8ce753b245b061
I20260927 21:11:10.905386 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0631bc0bd6984c54ae9964794df3a086. Latest consensus state: current_term: 1 leader_uuid: "0631bc0bd6984c54ae9964794df3a086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0631bc0bd6984c54ae9964794df3a086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } } }
I20260927 21:11:10.905493 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:10.905601 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0631bc0bd6984c54ae9964794df3a086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0631bc0bd6984c54ae9964794df3a086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } } }
I20260927 21:11:10.905696 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:10.906281 21341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:10.906816 21341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:10.910476 21341 catalog_manager.cc:1347] Generated new cluster ID: 4870c014ae954cea8492ae784761f18f
I20260927 21:11:10.910538 21341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:11:10 dist-test-slave-8bgs krb5kdc[21246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:11:10.915578 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:10.905579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:46755 (local address 127.0.0.1:43231)
0927 21:11:10.906239 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:10.906250 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:10.906285 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:11:10.906409 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:10.906418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:10.906484 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:10.906649 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:10.907457 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:10.908471 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:10.909225 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:10.909671 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:10.909787 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:10.912283 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:10.912286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:10.912287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:10.912298 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:10.913984 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:10.914371 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:10.914373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:10.914374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:10.914473 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:10.914711 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:10.914712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:10.914713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:10.914968 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:10.915162 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:10.915279 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:11:10.915381 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":67,"threads_started":1}
I20260927 21:11:10.916438 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:10.929145 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:10.919361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:58471 (local address 127.0.0.1:43231)
0927 21:11:10.919395 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:10.919397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:10.923285 (+ 3888us) server_negotiation.cc:407] Connection header received
0927 21:11:10.923358 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:10.923359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:10.923368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:10.923421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:10.923916 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:10.924387 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:10.925065 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:10.925266 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:10.925347 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:10.926493 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:10.926494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:10.926494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:10.926501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:10.927953 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:10.928324 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:10.928325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:10.928326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:10.928377 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:10.928616 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:10.928617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:10.928618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:10.928727 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:10.928835 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:10.928958 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:11:10.929027 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:11:10.934089 21341 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:10.934729 21341 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:10.942996 21341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0631bc0bd6984c54ae9964794df3a086: Generated new TSK 0
I20260927 21:11:10.943225 21341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:10.948698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21259
W20260927 21:11:10.952903 21244 connection.cc:535] client connection to 127.0.0.1:43231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43231): Connection reset by peer (error 104)
2026-09-27T21:11:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:10 dist-test-slave-8bgs krb5kdc[21358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:10 dist-test-slave-8bgs krb5kdc[21358](info): set up 2 sockets
Sep 27 21:11:10 dist-test-slave-8bgs krb5kdc[21358](info): commencing operation
krb5kdc: starting...
W20260927 21:11:11.979235 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:12 dist-test-slave-8bgs krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:12.092171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:12.171303 21371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:12.171528 21371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:12.171568 21371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:12.172717 21371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:12.172780 21371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:12.172806 21371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:12.172837 21371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:12.172861 21371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:12.174274 21371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:12.174549 21371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:12.174767 21371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:12.179298 21377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:12.179287 21376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:12.179275 21379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:12.179509 21371 server_base.cc:1034] running on GCE node
I20260927 21:11:12.179744 21371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:12.180055 21371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:12.181213 21371 hybrid_clock.cc:648] HybridClock initialized: now 1790543472181193 us; error 35 us; skew 500 ppm
Sep 27 21:11:12 dist-test-slave-8bgs krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:12.183588 21371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:12.183974 21371 webserver.cc:460] Webserver started at http://127.0.0.1:43677/ using document root <none> and password file <none>
I20260927 21:11:12.184180 21371 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:12.184242 21371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:12.184383 21371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:12.185346 21371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "a5e26aec94c64882bf92c48d7831a132"
format_stamp: "Formatted at 2026-09-27 21:11:12 on dist-test-slave-8bgs"
server_key: "068ec6d0e25d77abcde78952be6efd00"
server_key_iv: "d98e56ecdabf13ffcd5d922da04e1652"
server_key_version: "encryptionkey@0"
I20260927 21:11:12.185698 21371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5e26aec94c64882bf92c48d7831a132"
format_stamp: "Formatted at 2026-09-27 21:11:12 on dist-test-slave-8bgs"
server_key: "068ec6d0e25d77abcde78952be6efd00"
server_key_iv: "d98e56ecdabf13ffcd5d922da04e1652"
server_key_version: "encryptionkey@0"
I20260927 21:11:12.187044 21371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:12.187738 21386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:12.187889 21371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:12.187985 21371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "a5e26aec94c64882bf92c48d7831a132"
format_stamp: "Formatted at 2026-09-27 21:11:12 on dist-test-slave-8bgs"
server_key: "068ec6d0e25d77abcde78952be6efd00"
server_key_iv: "d98e56ecdabf13ffcd5d922da04e1652"
server_key_version: "encryptionkey@0"
I20260927 21:11:12.188058 21371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:12.204138 21371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:12.206467 21371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:12.206619 21371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:12.210543 21371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37271
I20260927 21:11:12.210608 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37271 every 8 connection(s)
I20260927 21:11:12.210947 21371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:12.211977 21439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:12.214237 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132: Bootstrap starting.
I20260927 21:11:12.214923 21439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:12.215276 21439 log.cc:824] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:12.215929 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132: No bootstrap required, opened a new log
I20260927 21:11:12.216584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21371
I20260927 21:11:12.216709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:12.216908 392 external_mini_cluster.cc:1428] Setting key 2ca4ecfac8775d81e7cda3789444d72a
I20260927 21:11:12.217934 21439 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e26aec94c64882bf92c48d7831a132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 21:11:12.218108 21439 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:12.218148 21439 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e26aec94c64882bf92c48d7831a132, State: Initialized, Role: FOLLOWER
I20260927 21:11:12.218281 21439 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e26aec94c64882bf92c48d7831a132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 21:11:12.218370 21439 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:12.218416 21439 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:12.218468 21439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:12.218982 21439 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e26aec94c64882bf92c48d7831a132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 21:11:12.219064 21439 leader_election.cc:302] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e26aec94c64882bf92c48d7831a132; no voters:
I20260927 21:11:12.219287 21439 leader_election.cc:288] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:12.219316 21444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:12.219419 21444 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [term 1 LEADER]: Becoming Leader. State: Replica: a5e26aec94c64882bf92c48d7831a132, State: Running, Role: LEADER
I20260927 21:11:12.219540 21439 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:12.219551 21444 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e26aec94c64882bf92c48d7831a132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260927 21:11:12.219962 21445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5e26aec94c64882bf92c48d7831a132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e26aec94c64882bf92c48d7831a132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260927 21:11:12.220064 21445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:12.220553 21451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:12.221019 21451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:12.221072 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e26aec94c64882bf92c48d7831a132. Latest consensus state: current_term: 1 leader_uuid: "a5e26aec94c64882bf92c48d7831a132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e26aec94c64882bf92c48d7831a132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260927 21:11:12.221138 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:12.222608 21451 catalog_manager.cc:1347] Generated new cluster ID: 421511d57356487ba7b5419070beb4a7
I20260927 21:11:12.222680 21451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:12.223299 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:12.217395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:39629 (local address 127.0.0.1:37271)
0927 21:11:12.217589 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:12.217596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:12.217620 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:11:12.217894 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:12.217899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:12.217948 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:12.218054 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:12.218346 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:12.219785 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:12.221081 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:12.221752 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:12.221866 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:12.222537 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:12.222540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:12.222541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:12.222750 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:12.222787 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:12.222803 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:12.222895 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:12.223058 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:11:12.223103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260927 21:11:12.224229 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:12.228986 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:12.225618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:56681 (local address 127.0.0.1:37271)
0927 21:11:12.225651 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:12.225653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:12.225794 (+ 141us) server_negotiation.cc:407] Connection header received
0927 21:11:12.225868 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:12.225869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:12.225878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:12.225902 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:12.226087 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:12.226764 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:12.227728 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:12.228056 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:12.228089 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:12.228753 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:12.228754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:12.228754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:12.228758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:12.228770 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:12.228774 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:12.228804 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:12.228913 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:11:12.228923 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:11:12.230860 21451 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:12.231421 21451 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:12.237192 21451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5e26aec94c64882bf92c48d7831a132: Generated new TSK 0
I20260927 21:11:12.237353 21451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:12.242184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21371
W20260927 21:11:12.246208 21356 connection.cc:535] client connection to 127.0.0.1:37271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37271): Connection reset by peer (error 104)
2026-09-27T21:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:12 dist-test-slave-8bgs krb5kdc[21470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:12 dist-test-slave-8bgs krb5kdc[21470](info): set up 2 sockets
Sep 27 21:11:12 dist-test-slave-8bgs krb5kdc[21470](info): commencing operation
krb5kdc: starting...
W20260927 21:11:13.258993 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:13 dist-test-slave-8bgs krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:13.370280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:13.449208 21483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:13.449443 21483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:13.449481 21483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:13.450639 21483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:13.450718 21483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:13.450743 21483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:13.450773 21483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:13.450805 21483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:13.452291 21483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:13.452574 21483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:13.452811 21483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:13.457602 21488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:13.457587 21491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:13.457659 21489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:13.457755 21483 server_base.cc:1034] running on GCE node
I20260927 21:11:13.458039 21483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:13.458318 21483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:13.459460 21483 hybrid_clock.cc:648] HybridClock initialized: now 1790543473459446 us; error 28 us; skew 500 ppm
Sep 27 21:11:13 dist-test-slave-8bgs krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:13.461967 21483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:13.462412 21483 webserver.cc:460] Webserver started at http://127.0.0.1:35689/ using document root <none> and password file <none>
I20260927 21:11:13.462651 21483 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:13.462723 21483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:13.462870 21483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:13.463984 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "1a56c3bd269f420f97fb562329e152f4"
format_stamp: "Formatted at 2026-09-27 21:11:13 on dist-test-slave-8bgs"
server_key: "1eada0cf29e3092a4ad283eaac101b30"
server_key_iv: "22c6e0004f46206d9f64f0fd7b61d7cd"
server_key_version: "encryptionkey@0"
I20260927 21:11:13.464380 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a56c3bd269f420f97fb562329e152f4"
format_stamp: "Formatted at 2026-09-27 21:11:13 on dist-test-slave-8bgs"
server_key: "1eada0cf29e3092a4ad283eaac101b30"
server_key_iv: "22c6e0004f46206d9f64f0fd7b61d7cd"
server_key_version: "encryptionkey@0"
I20260927 21:11:13.465665 21483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:11:13.466421 21498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:13.466583 21483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:13.466684 21483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "1a56c3bd269f420f97fb562329e152f4"
format_stamp: "Formatted at 2026-09-27 21:11:13 on dist-test-slave-8bgs"
server_key: "1eada0cf29e3092a4ad283eaac101b30"
server_key_iv: "22c6e0004f46206d9f64f0fd7b61d7cd"
server_key_version: "encryptionkey@0"
I20260927 21:11:13.466769 21483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:13.471628 21483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:13.474733 21483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:13.474923 21483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:13.480068 21483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42325
I20260927 21:11:13.480136 21550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42325 every 8 connection(s)
I20260927 21:11:13.480456 21483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:13.481498 21551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:13.483739 21551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4: Bootstrap starting.
I20260927 21:11:13.484290 21551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:13.484558 21551 log.cc:824] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:13.484589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21483
I20260927 21:11:13.484759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:13.484997 392 external_mini_cluster.cc:1428] Setting key 34878ae503c9230060f8a9c0863a311a
I20260927 21:11:13.485276 21551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4: No bootstrap required, opened a new log
I20260927 21:11:13.487035 21551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a56c3bd269f420f97fb562329e152f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260927 21:11:13.487244 21551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:13.487329 21551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a56c3bd269f420f97fb562329e152f4, State: Initialized, Role: FOLLOWER
I20260927 21:11:13.487478 21551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a56c3bd269f420f97fb562329e152f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260927 21:11:13.487581 21551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:13.487615 21551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:13.487661 21551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:13.488432 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:13.485512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:34345 (local address 127.0.0.1:42325)
0927 21:11:13.485680 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:13.485688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:13.485776 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:11:13.487024 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:13.487031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:13.487058 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:13.487268 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:13.487934 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:13.487937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:13.487937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:13.487940 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:13.487994 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:13.488005 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:13.488060 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:13.488169 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:11:13.488228 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260927 21:11:13.488725 21551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a56c3bd269f420f97fb562329e152f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260927 21:11:13.488852 21551 leader_election.cc:302] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a56c3bd269f420f97fb562329e152f4; no voters:
I20260927 21:11:13.489012 21551 leader_election.cc:288] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:13.489037 21556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:13.489223 21556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1a56c3bd269f420f97fb562329e152f4, State: Running, Role: LEADER
I20260927 21:11:13.489303 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:13.489313 21551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:13.489334 21556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a56c3bd269f420f97fb562329e152f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260927 21:11:13.489635 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a56c3bd269f420f97fb562329e152f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a56c3bd269f420f97fb562329e152f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } } }
I20260927 21:11:13.489737 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:13.489895 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a56c3bd269f420f97fb562329e152f4. Latest consensus state: current_term: 1 leader_uuid: "1a56c3bd269f420f97fb562329e152f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a56c3bd269f420f97fb562329e152f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } } }
I20260927 21:11:13.489977 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:13.490394 21563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:13.490916 21563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:13.492466 21563 catalog_manager.cc:1347] Generated new cluster ID: 013065fce42f48aaa9e175d7c08620df
I20260927 21:11:13.492518 21563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:13.493218 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:13.491807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:56889 (local address 127.0.0.1:42325)
0927 21:11:13.491851 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:13.491853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:13.492228 (+ 375us) server_negotiation.cc:407] Connection header received
0927 21:11:13.492322 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:13.492322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:13.492327 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:13.492356 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:13.492999 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:13.493000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:13.493001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:13.493005 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:13.493020 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:13.493025 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:13.493086 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:13.493147 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:11:13.493161 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:13.504129 21563 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:13.504261 21563 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:13.508455 21563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a56c3bd269f420f97fb562329e152f4: Generated new TSK 0
I20260927 21:11:13.508594 21563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:13.514587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21483
2026-09-27T21:11:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:13 dist-test-slave-8bgs krb5kdc[21582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:13 dist-test-slave-8bgs krb5kdc[21582](info): set up 2 sockets
Sep 27 21:11:13 dist-test-slave-8bgs krb5kdc[21582](info): commencing operation
krb5kdc: starting...
W20260927 21:11:14.539280 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:14 dist-test-slave-8bgs krb5kdc[21582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:14.651858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:14.730177 21595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:14.730441 21595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:14.730480 21595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:14.731668 21595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:14.731743 21595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:14.731768 21595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:14.731791 21595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:14.731818 21595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:14.733278 21595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:14.733568 21595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:14.733793 21595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:11:14.738571 21595 server_base.cc:1034] running on GCE node
W20260927 21:11:14.738725 21601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:14.738698 21600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:14.738713 21603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:14.739073 21595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:14.739406 21595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:14.740566 21595 hybrid_clock.cc:648] HybridClock initialized: now 1790543474740562 us; error 32 us; skew 500 ppm
Sep 27 21:11:14 dist-test-slave-8bgs krb5kdc[21582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:14.742862 21595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:14.743345 21595 webserver.cc:460] Webserver started at http://127.0.0.1:36709/ using document root <none> and password file <none>
I20260927 21:11:14.743563 21595 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:14.743630 21595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:14.743782 21595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:14.744832 21595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "788e1b4136ef48a1a825263a35e75c01"
format_stamp: "Formatted at 2026-09-27 21:11:14 on dist-test-slave-8bgs"
server_key: "c8244e7ade2c4531821375b29a1a68ed"
server_key_iv: "0dbc940888e2de376c9c28ef50b403a9"
server_key_version: "encryptionkey@0"
I20260927 21:11:14.745196 21595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "788e1b4136ef48a1a825263a35e75c01"
format_stamp: "Formatted at 2026-09-27 21:11:14 on dist-test-slave-8bgs"
server_key: "c8244e7ade2c4531821375b29a1a68ed"
server_key_iv: "0dbc940888e2de376c9c28ef50b403a9"
server_key_version: "encryptionkey@0"
I20260927 21:11:14.746505 21595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:11:14.747259 21610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:14.747557 21595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:11:14.747658 21595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "788e1b4136ef48a1a825263a35e75c01"
format_stamp: "Formatted at 2026-09-27 21:11:14 on dist-test-slave-8bgs"
server_key: "c8244e7ade2c4531821375b29a1a68ed"
server_key_iv: "0dbc940888e2de376c9c28ef50b403a9"
server_key_version: "encryptionkey@0"
I20260927 21:11:14.747740 21595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:14.758735 21595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:14.760478 21595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:14.760638 21595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:14.764582 21595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45667
I20260927 21:11:14.764614 21662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45667 every 8 connection(s)
I20260927 21:11:14.764948 21595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:14.765728 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:14.766060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21595
I20260927 21:11:14.766188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:14.766400 392 external_mini_cluster.cc:1428] Setting key e20e6450f4066f1ba8395f98b03042c7
I20260927 21:11:14.768800 21663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01: Bootstrap starting.
I20260927 21:11:14.769381 21663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:14.769670 21663 log.cc:824] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:14.770349 21663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01: No bootstrap required, opened a new log
I20260927 21:11:14.772636 21663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e1b4136ef48a1a825263a35e75c01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260927 21:11:14.772801 21663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:14.772835 21663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788e1b4136ef48a1a825263a35e75c01, State: Initialized, Role: FOLLOWER
Sep 27 21:11:14 dist-test-slave-8bgs krb5kdc[21582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:11:14.772954 21663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e1b4136ef48a1a825263a35e75c01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260927 21:11:14.773041 21663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:14.773064 21663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:14.773132 21663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:14.773789 21663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e1b4136ef48a1a825263a35e75c01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260927 21:11:14.773890 21663 leader_election.cc:302] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788e1b4136ef48a1a825263a35e75c01; no voters:
I20260927 21:11:14.774055 21663 leader_election.cc:288] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:14.774116 21668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:14.774258 21668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [term 1 LEADER]: Becoming Leader. State: Replica: 788e1b4136ef48a1a825263a35e75c01, State: Running, Role: LEADER
I20260927 21:11:14.774308 21663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:14.774722 21668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e1b4136ef48a1a825263a35e75c01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } }
I20260927 21:11:14.775213 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788e1b4136ef48a1a825263a35e75c01. Latest consensus state: current_term: 1 leader_uuid: "788e1b4136ef48a1a825263a35e75c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e1b4136ef48a1a825263a35e75c01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } } }
I20260927 21:11:14.775286 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:14.775652 21669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788e1b4136ef48a1a825263a35e75c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e1b4136ef48a1a825263a35e75c01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45667 } } }
I20260927 21:11:14.775803 21669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01 [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:14.776228 21684 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:14.776281 21684 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:14.776331 21678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:14.776508 21678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:14.777365 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:14.766873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:46137 (local address 127.0.0.1:45667)
0927 21:11:14.767050 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:14.767057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:14.767080 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:11:14.767538 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:14.767544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:14.767597 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:14.767710 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:14.768501 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:14.769416 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:14.770244 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:14.770830 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:14.770949 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:14.773495 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:14.773499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:14.773499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:14.773511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:14.775680 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:14.776214 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:14.776216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:14.776217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:14.776325 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:14.776590 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:14.776591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:14.776591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:14.776815 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:14.776983 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:14.777106 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:11:14.777204 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260927 21:11:14.777984 21678 catalog_manager.cc:1347] Generated new cluster ID: 16e24f1687704ed3ad5e1e1dc32f2183
I20260927 21:11:14.778036 21678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:14.779570 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:14.789578 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:14.783425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:34481 (local address 127.0.0.1:45667)
0927 21:11:14.783464 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:14.783466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:14.783611 (+ 145us) server_negotiation.cc:407] Connection header received
0927 21:11:14.783683 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:14.783684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:14.783694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:14.783749 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:14.784250 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:14.784722 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:14.785428 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:14.785657 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:14.785735 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:14.786910 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:14.786912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:14.786912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:14.786919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:14.788370 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:14.788750 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:14.788751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:14.788751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:14.788811 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:14.789039 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:14.789040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:14.789041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:14.789153 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:14.789275 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:14.789395 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:11:14.789463 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:11:14.806069 21678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:14.806777 21678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:14.818437 21678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788e1b4136ef48a1a825263a35e75c01: Generated new TSK 0
I20260927 21:11:14.818656 21678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:14.824769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21595
W20260927 21:11:14.829114 21580 connection.cc:535] client connection to 127.0.0.1:45667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45667): Connection reset by peer (error 104)
2026-09-27T21:11:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:14 dist-test-slave-8bgs krb5kdc[21694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:14 dist-test-slave-8bgs krb5kdc[21694](info): set up 2 sockets
Sep 27 21:11:14 dist-test-slave-8bgs krb5kdc[21694](info): commencing operation
krb5kdc: starting...
W20260927 21:11:15.863133 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:15 dist-test-slave-8bgs krb5kdc[21694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:15.977640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:16.056628 21707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:16.056833 21707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:16.056856 21707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:16.058007 21707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:16.058048 21707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:16.058063 21707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:16.058101 21707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:16.058131 21707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:16.059602 21707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:16.059855 21707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:16.060082 21707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:11:16.064960 21707 server_base.cc:1034] running on GCE node
W20260927 21:11:16.064993 21712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:16.065107 21713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:16.064988 21715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:16.065410 21707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:16.065685 21707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:16.066826 21707 hybrid_clock.cc:648] HybridClock initialized: now 1790543476066819 us; error 30 us; skew 500 ppm
Sep 27 21:11:16 dist-test-slave-8bgs krb5kdc[21694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:16.075495 21707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:16.075963 21707 webserver.cc:460] Webserver started at http://127.0.0.1:44773/ using document root <none> and password file <none>
I20260927 21:11:16.076200 21707 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:16.076268 21707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:16.076444 21707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:16.077471 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "a5ef85b1f0de43ba9a1e721afd55d441"
format_stamp: "Formatted at 2026-09-27 21:11:16 on dist-test-slave-8bgs"
server_key: "1643e565c884a02fa54b52df3aa0fda0"
server_key_iv: "2f2bf2fbaf2cfdf1e99f6b763a800c94"
server_key_version: "encryptionkey@0"
I20260927 21:11:16.077834 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5ef85b1f0de43ba9a1e721afd55d441"
format_stamp: "Formatted at 2026-09-27 21:11:16 on dist-test-slave-8bgs"
server_key: "1643e565c884a02fa54b52df3aa0fda0"
server_key_iv: "2f2bf2fbaf2cfdf1e99f6b763a800c94"
server_key_version: "encryptionkey@0"
I20260927 21:11:16.079144 21707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:11:16.079873 21722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:16.080046 21707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:16.080143 21707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "a5ef85b1f0de43ba9a1e721afd55d441"
format_stamp: "Formatted at 2026-09-27 21:11:16 on dist-test-slave-8bgs"
server_key: "1643e565c884a02fa54b52df3aa0fda0"
server_key_iv: "2f2bf2fbaf2cfdf1e99f6b763a800c94"
server_key_version: "encryptionkey@0"
I20260927 21:11:16.080226 21707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:16.095623 21707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:16.097345 21707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:16.097501 21707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:16.101449 21707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38451
I20260927 21:11:16.101480 21774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38451 every 8 connection(s)
I20260927 21:11:16.101815 21707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:16.102726 21775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:16.104959 21775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441: Bootstrap starting.
I20260927 21:11:16.105458 21775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:16.105688 21775 log.cc:824] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:16.106338 21775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441: No bootstrap required, opened a new log
I20260927 21:11:16.107849 21775 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260927 21:11:16.107990 21775 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:16.108013 21775 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ef85b1f0de43ba9a1e721afd55d441, State: Initialized, Role: FOLLOWER
I20260927 21:11:16.108142 21775 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260927 21:11:16.108222 21775 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:16.108245 21775 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:16.108274 21775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:16.108731 21775 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260927 21:11:16.108805 21775 leader_election.cc:302] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5ef85b1f0de43ba9a1e721afd55d441; no voters:
I20260927 21:11:16.108920 21775 leader_election.cc:288] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:16.108978 21778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:16.109141 21778 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [term 1 LEADER]: Becoming Leader. State: Replica: a5ef85b1f0de43ba9a1e721afd55d441, State: Running, Role: LEADER
I20260927 21:11:16.109175 21775 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:16.109248 21778 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260927 21:11:16.109673 21780 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ef85b1f0de43ba9a1e721afd55d441. Latest consensus state: current_term: 1 leader_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } } }
I20260927 21:11:16.109752 21780 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:16.109789 21779 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ef85b1f0de43ba9a1e721afd55d441" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } } }
I20260927 21:11:16.109870 21779 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:16.110296 21786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:16.110800 21786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:16.112028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21707
I20260927 21:11:16.112143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:16.112351 392 external_mini_cluster.cc:1428] Setting key 3c69cf4fe2ae8a058f6178f5108ad78a
I20260927 21:11:16.112566 21786 catalog_manager.cc:1347] Generated new cluster ID: f2e88747813b449fbb200274530163a7
I20260927 21:11:16.112618 21786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:16.118515 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:16.112965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:41889 (local address 127.0.0.1:38451)
0927 21:11:16.113118 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:16.113126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:16.113153 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:11:16.113272 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:16.113279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:16.113331 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:16.113475 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:16.113728 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:16.115083 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:16.116170 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:16.117039 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:16.117157 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:16.117854 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:16.117858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:16.117859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:16.118064 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:16.118106 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:16.118120 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:16.118204 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:16.118293 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 21:11:16.118318 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260927 21:11:16.119444 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:16.126601 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:16.120793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:40837 (local address 127.0.0.1:38451)
0927 21:11:16.120823 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:16.120824 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:16.123283 (+ 2459us) server_negotiation.cc:407] Connection header received
0927 21:11:16.123339 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:16.123340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:16.123351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:16.123406 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:16.123523 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:16.124247 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:16.125271 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:16.125583 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:16.125635 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:16.126339 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:16.126340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:16.126340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:16.126344 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:16.126359 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:16.126363 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:16.126427 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:16.126523 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 21:11:16.126533 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:11:16.131510 21786 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:16.131982 21786 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:16.141305 21786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5ef85b1f0de43ba9a1e721afd55d441: Generated new TSK 0
I20260927 21:11:16.141522 21786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:16.144935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21707
W20260927 21:11:16.149127 21692 connection.cc:535] client connection to 127.0.0.1:38451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38451): Connection reset by peer (error 104)
2026-09-27T21:11:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:16 dist-test-slave-8bgs krb5kdc[21806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:16 dist-test-slave-8bgs krb5kdc[21806](info): set up 2 sockets
Sep 27 21:11:16 dist-test-slave-8bgs krb5kdc[21806](info): commencing operation
krb5kdc: starting...
W20260927 21:11:17.159225 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:17 dist-test-slave-8bgs krb5kdc[21806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:11:17.271704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:17.351492 21819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:17.351742 21819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:17.351783 21819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:17.352916 21819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:17.352989 21819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:17.353013 21819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:17.353034 21819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:17.353065 21819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:17.354512 21819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:17.354785 21819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:17.355007 21819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:17.359622 21824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:17.359632 21827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:17.359778 21819 server_base.cc:1034] running on GCE node
W20260927 21:11:17.359632 21825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:17.360036 21819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:17.360316 21819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:17.361485 21819 hybrid_clock.cc:648] HybridClock initialized: now 1790543477361476 us; error 41 us; skew 500 ppm
Sep 27 21:11:17 dist-test-slave-8bgs krb5kdc[21806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:17.364032 21819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:11:17.364463 21819 webserver.cc:460] Webserver started at http://127.0.0.1:42547/ using document root <none> and password file <none>
I20260927 21:11:17.364697 21819 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:17.364765 21819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:17.364914 21819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:17.365935 21819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "a2635e51fdb540729d1bfd2c9ae408f1"
format_stamp: "Formatted at 2026-09-27 21:11:17 on dist-test-slave-8bgs"
server_key: "e31997e18e5632f77edc3a40cc2ef3dd"
server_key_iv: "95de69e652ed5ac951e8dcf1d95f59d1"
server_key_version: "encryptionkey@0"
I20260927 21:11:17.366303 21819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2635e51fdb540729d1bfd2c9ae408f1"
format_stamp: "Formatted at 2026-09-27 21:11:17 on dist-test-slave-8bgs"
server_key: "e31997e18e5632f77edc3a40cc2ef3dd"
server_key_iv: "95de69e652ed5ac951e8dcf1d95f59d1"
server_key_version: "encryptionkey@0"
I20260927 21:11:17.367640 21819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:11:17.368335 21834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:17.368566 21819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:17.368673 21819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "a2635e51fdb540729d1bfd2c9ae408f1"
format_stamp: "Formatted at 2026-09-27 21:11:17 on dist-test-slave-8bgs"
server_key: "e31997e18e5632f77edc3a40cc2ef3dd"
server_key_iv: "95de69e652ed5ac951e8dcf1d95f59d1"
server_key_version: "encryptionkey@0"
I20260927 21:11:17.368745 21819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:17.371944 21819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:17.373677 21819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:17.373823 21819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:17.377467 21819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45069
I20260927 21:11:17.377511 21886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45069 every 8 connection(s)
I20260927 21:11:17.377820 21819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:17.378731 21887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:17.380864 21887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1: Bootstrap starting.
I20260927 21:11:17.381397 21887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:17.381669 21887 log.cc:824] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:17.382289 21887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1: No bootstrap required, opened a new log
I20260927 21:11:17.383898 21887 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260927 21:11:17.384028 21887 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:17.384050 21887 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2635e51fdb540729d1bfd2c9ae408f1, State: Initialized, Role: FOLLOWER
I20260927 21:11:17.384253 21887 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260927 21:11:17.384331 21887 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:17.384378 21887 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:17.384426 21887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:17.384922 21887 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260927 21:11:17.385033 21887 leader_election.cc:302] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2635e51fdb540729d1bfd2c9ae408f1; no voters:
I20260927 21:11:17.385200 21887 leader_election.cc:288] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:17.385254 21890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:17.385401 21890 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [term 1 LEADER]: Becoming Leader. State: Replica: a2635e51fdb540729d1bfd2c9ae408f1, State: Running, Role: LEADER
I20260927 21:11:17.385437 21887 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:17.385509 21890 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260927 21:11:17.385852 21892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2635e51fdb540729d1bfd2c9ae408f1. Latest consensus state: current_term: 1 leader_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } } }
I20260927 21:11:17.385952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21819
I20260927 21:11:17.385973 21891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2635e51fdb540729d1bfd2c9ae408f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } } }
I20260927 21:11:17.385983 21892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:17.386039 21891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:17.386090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:17.386286 21898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:17.386312 392 external_mini_cluster.cc:1428] Setting key c933bdcba47c18dd54f6106ae604d9f7
I20260927 21:11:17.386976 21898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:17.388751 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:17.386894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:55939 (local address 127.0.0.1:45069)
0927 21:11:17.387056 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:17.387065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:17.387274 (+ 209us) server_negotiation.cc:407] Connection header received
0927 21:11:17.387391 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:17.387399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:17.387431 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:17.387582 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:17.388321 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:17.388325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:17.388325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:17.388329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:17.388365 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:17.388378 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:17.388419 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:17.388554 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:11:17.388598 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 21:11:17.389720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:17.392071 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:17.390986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.102:43573 (local address 127.0.0.1:45069)
0927 21:11:17.391017 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:17.391019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:17.391145 (+ 126us) server_negotiation.cc:407] Connection header received
0927 21:11:17.391217 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:17.391218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:17.391223 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:17.391277 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:17.391848 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:17.391849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:17.391849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:17.391853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:17.391866 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:17.391870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:17.391925 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:17.391985 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:11:17.391998 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 21:11:17.395473 21898 catalog_manager.cc:1347] Generated new cluster ID: 7a6a8ff3300f4dd184da679d83f38d35
I20260927 21:11:17.395533 21898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:17.409197 21898 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:17.409408 21898 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:17.416908 21898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2635e51fdb540729d1bfd2c9ae408f1: Generated new TSK 0
I20260927 21:11:17.417102 21898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:17.419962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21819
2026-09-27T21:11:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:17 dist-test-slave-8bgs krb5kdc[21918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:17 dist-test-slave-8bgs krb5kdc[21918](info): set up 2 sockets
Sep 27 21:11:17 dist-test-slave-8bgs krb5kdc[21918](info): commencing operation
krb5kdc: starting...
W20260927 21:11:18.467276 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:18 dist-test-slave-8bgs krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:18.580888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:18.659880 21931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:18.660077 21931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:18.660099 21931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:18.661144 21931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:18.661176 21931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:18.661190 21931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:18.661203 21931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:18.661216 21931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:18.662568 21931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:18.662756 21931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:18.662912 21931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:18.667747 21937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:18.667732 21936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:18.667785 21931 server_base.cc:1034] running on GCE node
W20260927 21:11:18.667732 21939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:18.668182 21931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:18.668478 21931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:18.669647 21931 hybrid_clock.cc:648] HybridClock initialized: now 1790543478669642 us; error 37 us; skew 500 ppm
Sep 27 21:11:18 dist-test-slave-8bgs krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:18.672127 21931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:18.672497 21931 webserver.cc:460] Webserver started at http://127.0.98.62:44021/ using document root <none> and password file <none>
I20260927 21:11:18.672681 21931 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:18.672721 21931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:18.672804 21931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:18.673763 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "75f1d3e82ad249b5a05b7eb68a5775a2"
format_stamp: "Formatted at 2026-09-27 21:11:18 on dist-test-slave-8bgs"
server_key: "580389253f141606e18bdcc98f1b1870"
server_key_iv: "391d20c6eff8be643d77d07029e517b8"
server_key_version: "encryptionkey@0"
I20260927 21:11:18.674060 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "75f1d3e82ad249b5a05b7eb68a5775a2"
format_stamp: "Formatted at 2026-09-27 21:11:18 on dist-test-slave-8bgs"
server_key: "580389253f141606e18bdcc98f1b1870"
server_key_iv: "391d20c6eff8be643d77d07029e517b8"
server_key_version: "encryptionkey@0"
I20260927 21:11:18.675333 21931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:11:18.676028 21946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:18.676210 21931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:18.676292 21931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "75f1d3e82ad249b5a05b7eb68a5775a2"
format_stamp: "Formatted at 2026-09-27 21:11:18 on dist-test-slave-8bgs"
server_key: "580389253f141606e18bdcc98f1b1870"
server_key_iv: "391d20c6eff8be643d77d07029e517b8"
server_key_version: "encryptionkey@0"
I20260927 21:11:18.676344 21931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:18.684719 21931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:18.686458 21931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:18.686573 21931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:18.690204 21931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44815
I20260927 21:11:18.690274 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44815 every 8 connection(s)
I20260927 21:11:18.690619 21931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:18.691588 21999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:18.693746 21999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2: Bootstrap starting.
I20260927 21:11:18.694293 21999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:18.694568 21999 log.cc:824] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:18.694943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 21931
I20260927 21:11:18.695050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:18.695245 392 external_mini_cluster.cc:1428] Setting key 7229a30f153e3c2ccba1f6e3a531325a
I20260927 21:11:18.695390 21999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2: No bootstrap required, opened a new log
I20260927 21:11:18.697221 21999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44815 } }
I20260927 21:11:18.697369 21999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:18.697419 21999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f1d3e82ad249b5a05b7eb68a5775a2, State: Initialized, Role: FOLLOWER
I20260927 21:11:18.697548 21999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44815 } }
I20260927 21:11:18.697623 21999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:18.697646 21999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:18.697696 21999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:18.698204 21999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44815 } }
I20260927 21:11:18.698285 21999 leader_election.cc:302] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75f1d3e82ad249b5a05b7eb68a5775a2; no voters:
I20260927 21:11:18.698463 21999 leader_election.cc:288] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:18.698498 22004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:18.698607 22004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [term 1 LEADER]: Becoming Leader. State: Replica: 75f1d3e82ad249b5a05b7eb68a5775a2, State: Running, Role: LEADER
I20260927 21:11:18.698714 21999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:18.698737 22004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44815 } }
I20260927 21:11:18.699661 22005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75f1d3e82ad249b5a05b7eb68a5775a2. Latest consensus state: current_term: 1 leader_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44815 } } }
I20260927 21:11:18.699728 22005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:18.699599 22006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f1d3e82ad249b5a05b7eb68a5775a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44815 } } }
I20260927 21:11:18.699918 22015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:18.699934 22006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:18.700529 22015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:18.701932 22015 catalog_manager.cc:1347] Generated new cluster ID: c32c37cf48f044e9a5425b274601aedc
I20260927 21:11:18.701993 22015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:11:18 dist-test-slave-8bgs krb5kdc[21918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:18.706377 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:18.695753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53213 (local address 127.0.98.62:44815)
0927 21:11:18.695930 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:18.695939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:18.695973 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:11:18.696575 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:18.696582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:18.696628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:18.696772 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:18.697407 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:18.698804 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:18.699913 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:18.700483 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:18.700623 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:18.703115 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:18.703119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:18.703119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:18.703131 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:18.704781 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:18.705157 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:18.705160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:18.705160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:18.705252 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:18.705479 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:18.705480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:18.705481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:18.705706 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:18.705878 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:18.705963 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:11:18.706060 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260927 21:11:18.707264 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:18.721166 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:18.711367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35687 (local address 127.0.98.62:44815)
0927 21:11:18.711401 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:18.711402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:18.715368 (+ 3966us) server_negotiation.cc:407] Connection header received
0927 21:11:18.715443 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:18.715444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:18.715454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:18.715506 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:18.716022 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:18.716520 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:18.717211 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:18.717435 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:18.717507 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:18.718644 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:18.718646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:18.718646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:18.718653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:18.720037 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:18.720386 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:18.720387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:18.720388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:18.720442 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:18.720666 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:18.720667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:18.720667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:18.720773 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:18.720878 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:18.720996 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:11:18.721055 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:11:18.739377 22015 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:18.740054 22015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:18.745697 22015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75f1d3e82ad249b5a05b7eb68a5775a2: Generated new TSK 0
I20260927 21:11:18.745886 22015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:18.753986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 21931
W20260927 21:11:18.758339 21916 connection.cc:535] client connection to 127.0.98.62:44815 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44815): Connection reset by peer (error 104)
2026-09-27T21:11:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:18 dist-test-slave-8bgs krb5kdc[22030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:18 dist-test-slave-8bgs krb5kdc[22030](info): set up 2 sockets
Sep 27 21:11:18 dist-test-slave-8bgs krb5kdc[22030](info): commencing operation
krb5kdc: starting...
W20260927 21:11:19.795219 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:19 dist-test-slave-8bgs krb5kdc[22030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:19.909623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:19.989646 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:19.989846 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:19.989871 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:19.991012 22043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:19.991055 22043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:19.991070 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:19.991143 22043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:19.991165 22043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:19.992663 22043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:19.992892 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:19.993146 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:19.997709 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:19.997723 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:19.997716 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:19.997978 22043 server_base.cc:1034] running on GCE node
I20260927 21:11:19.998212 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:19.998499 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:19.999660 22043 hybrid_clock.cc:648] HybridClock initialized: now 1790543479999653 us; error 35 us; skew 500 ppm
Sep 27 21:11:19 dist-test-slave-8bgs krb5kdc[22030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:20.002063 22043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:20.002481 22043 webserver.cc:460] Webserver started at http://127.0.98.62:43853/ using document root <none> and password file <none>
I20260927 21:11:20.002722 22043 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:20.002790 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:20.002940 22043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:20.004103 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "03d268a81370435fa3cb919de800f9b5"
format_stamp: "Formatted at 2026-09-27 21:11:20 on dist-test-slave-8bgs"
server_key: "4bfba65738067e8de7cb2a9ea28e74ae"
server_key_iv: "642fcbce2cdac51e759deeb9202b5e42"
server_key_version: "encryptionkey@0"
I20260927 21:11:20.004467 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "03d268a81370435fa3cb919de800f9b5"
format_stamp: "Formatted at 2026-09-27 21:11:20 on dist-test-slave-8bgs"
server_key: "4bfba65738067e8de7cb2a9ea28e74ae"
server_key_iv: "642fcbce2cdac51e759deeb9202b5e42"
server_key_version: "encryptionkey@0"
I20260927 21:11:20.005788 22043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:20.006567 22058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:20.006740 22043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:20.006839 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "03d268a81370435fa3cb919de800f9b5"
format_stamp: "Formatted at 2026-09-27 21:11:20 on dist-test-slave-8bgs"
server_key: "4bfba65738067e8de7cb2a9ea28e74ae"
server_key_iv: "642fcbce2cdac51e759deeb9202b5e42"
server_key_version: "encryptionkey@0"
I20260927 21:11:20.006927 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:20.025434 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:20.027406 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:20.027575 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:20.031311 22043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37741
I20260927 21:11:20.031355 22110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37741 every 8 connection(s)
I20260927 21:11:20.031754 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:20.032673 22111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:20.034060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22043
I20260927 21:11:20.034155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:20.034351 392 external_mini_cluster.cc:1428] Setting key 61d18c7d122c54a7cde100b488a45e84
I20260927 21:11:20.035492 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5: Bootstrap starting.
I20260927 21:11:20.036204 22111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:20.036517 22111 log.cc:824] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:20.037246 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5: No bootstrap required, opened a new log
I20260927 21:11:20.039606 22111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d268a81370435fa3cb919de800f9b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 21:11:20.039796 22111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:20.039896 22111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d268a81370435fa3cb919de800f9b5, State: Initialized, Role: FOLLOWER
I20260927 21:11:20.040041 22111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d268a81370435fa3cb919de800f9b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 21:11:20.040149 22111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:20.040195 22111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:20.040246 22111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:20.040588 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:20.034916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46541 (local address 127.0.98.62:37741)
0927 21:11:20.035133 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:20.035143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:20.035178 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:11:20.035732 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:20.035741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:20.035807 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:20.035974 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:20.036145 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:20.037513 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:20.038287 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:20.038945 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:20.039078 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:20.039935 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:20.039939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:20.039939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:20.040146 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:20.040205 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:20.040221 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:20.040307 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:20.040393 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:11:20.040421 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260927 21:11:20.040998 22111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d268a81370435fa3cb919de800f9b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 21:11:20.041121 22111 leader_election.cc:302] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03d268a81370435fa3cb919de800f9b5; no voters:
I20260927 21:11:20.041293 22111 leader_election.cc:288] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:20.041317 22116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:20.041499 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:20.041592 22111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:20.042882 22116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [term 1 LEADER]: Becoming Leader. State: Replica: 03d268a81370435fa3cb919de800f9b5, State: Running, Role: LEADER
I20260927 21:11:20.042980 22116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d268a81370435fa3cb919de800f9b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } }
I20260927 21:11:20.043568 22118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03d268a81370435fa3cb919de800f9b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d268a81370435fa3cb919de800f9b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } } }
I20260927 21:11:20.043805 22118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:20.043596 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03d268a81370435fa3cb919de800f9b5. Latest consensus state: current_term: 1 leader_uuid: "03d268a81370435fa3cb919de800f9b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d268a81370435fa3cb919de800f9b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37741 } } }
I20260927 21:11:20.044065 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5 [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:20.044315 22135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:20.044371 22135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:20.044459 22136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:20.044654 22136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:20.046579 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:20.043222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39707 (local address 127.0.98.62:37741)
0927 21:11:20.043253 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:20.043254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:20.043434 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:11:20.043466 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:20.043466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:20.043477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:20.043537 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:20.043862 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:20.044575 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:20.045310 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:20.045615 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:20.045702 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:20.046319 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:20.046320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:20.046321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:20.046325 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:20.046341 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:20.046346 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:20.046413 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:20.046493 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:11:20.046507 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 21:11:20.047731 22136 catalog_manager.cc:1347] Generated new cluster ID: 9a206316ba11478cbccd17daddc26eff
I20260927 21:11:20.047789 22136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:20.062177 22136 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:20.062949 22136 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:20.071503 22136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03d268a81370435fa3cb919de800f9b5: Generated new TSK 0
I20260927 21:11:20.071759 22136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:20.080484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22043
W20260927 21:11:20.084776 22028 connection.cc:535] client connection to 127.0.98.62:37741 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37741): Connection reset by peer (error 104)
2026-09-27T21:11:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:20 dist-test-slave-8bgs krb5kdc[22142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:20 dist-test-slave-8bgs krb5kdc[22142](info): set up 2 sockets
Sep 27 21:11:20 dist-test-slave-8bgs krb5kdc[22142](info): commencing operation
krb5kdc: starting...
W20260927 21:11:21.111305 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:21 dist-test-slave-8bgs krb5kdc[22142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:21.225828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:21.306206 22155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:21.306469 22155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:21.306542 22155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:21.307780 22155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:21.307853 22155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:21.307878 22155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:21.307900 22155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:21.307930 22155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:21.309551 22155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:21.309823 22155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:21.310063 22155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:21.314828 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:21.314896 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:21.314945 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:21.315145 22155 server_base.cc:1034] running on GCE node
I20260927 21:11:21.315356 22155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:21.315632 22155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:21.316773 22155 hybrid_clock.cc:648] HybridClock initialized: now 1790543481316763 us; error 32 us; skew 500 ppm
Sep 27 21:11:21 dist-test-slave-8bgs krb5kdc[22142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:21.323004 22155 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:21.323483 22155 webserver.cc:460] Webserver started at http://127.0.98.62:40353/ using document root <none> and password file <none>
I20260927 21:11:21.323665 22155 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:21.323712 22155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:21.323799 22155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:21.324812 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "6e7d2159706c42a6a8b2e4632490e9a9"
format_stamp: "Formatted at 2026-09-27 21:11:21 on dist-test-slave-8bgs"
server_key: "ce1fb560b6983138bb653c622e7da82f"
server_key_iv: "73d69afa5e16b4083fb4a7dce20d87ed"
server_key_version: "encryptionkey@0"
I20260927 21:11:21.325109 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e7d2159706c42a6a8b2e4632490e9a9"
format_stamp: "Formatted at 2026-09-27 21:11:21 on dist-test-slave-8bgs"
server_key: "ce1fb560b6983138bb653c622e7da82f"
server_key_iv: "73d69afa5e16b4083fb4a7dce20d87ed"
server_key_version: "encryptionkey@0"
I20260927 21:11:21.326344 22155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:11:21.327020 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:21.327212 22155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:21.327306 22155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "6e7d2159706c42a6a8b2e4632490e9a9"
format_stamp: "Formatted at 2026-09-27 21:11:21 on dist-test-slave-8bgs"
server_key: "ce1fb560b6983138bb653c622e7da82f"
server_key_iv: "73d69afa5e16b4083fb4a7dce20d87ed"
server_key_version: "encryptionkey@0"
I20260927 21:11:21.327358 22155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:21.330487 22155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:21.332304 22155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:21.332407 22155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:21.336054 22155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44575
I20260927 21:11:21.336105 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44575 every 8 connection(s)
I20260927 21:11:21.336434 22155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:21.337363 22223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:21.339599 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9: Bootstrap starting.
I20260927 21:11:21.340215 22223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:21.340224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22155
I20260927 21:11:21.340318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:21.340533 392 external_mini_cluster.cc:1428] Setting key e4359f4a9cb21b12914f164804578205
I20260927 21:11:21.340550 22223 log.cc:824] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:21.341249 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9: No bootstrap required, opened a new log
I20260927 21:11:21.343032 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:21.340944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58285 (local address 127.0.98.62:44575)
0927 21:11:21.341118 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:21.341128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:21.341164 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:11:21.341645 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:21.341652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:21.341678 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:21.341827 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:21.342489 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:21.342493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:21.342493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:21.342498 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:21.342542 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:21.342552 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:21.342644 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:21.342681 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 21:11:21.342720 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260927 21:11:21.343050 22223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260927 21:11:21.343261 22223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:21.343309 22223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7d2159706c42a6a8b2e4632490e9a9, State: Initialized, Role: FOLLOWER
I20260927 21:11:21.343436 22223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260927 21:11:21.343554 22223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:21.343637 22223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:21.343693 22223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:21.344027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:21.344293 22223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260927 21:11:21.344391 22223 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e7d2159706c42a6a8b2e4632490e9a9; no voters:
I20260927 21:11:21.344545 22223 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:21.344753 22223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:21.345468 22228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:21.345607 22228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7d2159706c42a6a8b2e4632490e9a9, State: Running, Role: LEADER
I20260927 21:11:21.345712 22228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260927 21:11:21.346040 22239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } } }
I20260927 21:11:21.346060 22228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7d2159706c42a6a8b2e4632490e9a9. Latest consensus state: current_term: 1 leader_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7d2159706c42a6a8b2e4632490e9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } } }
I20260927 21:11:21.346176 22239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:21.346238 22228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9 [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:21.346944 22246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:21.347124 22246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:21.347553 22248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:21.347724 22248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:21.347801 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:21.346382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54221 (local address 127.0.98.62:44575)
0927 21:11:21.346414 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:21.346416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:21.346818 (+ 402us) server_negotiation.cc:407] Connection header received
0927 21:11:21.346869 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:21.346869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:21.346874 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:21.346939 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:21.347551 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:21.347552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:21.347553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:21.347556 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:21.347571 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:21.347575 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:21.347632 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:21.347723 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:11:21.347744 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 21:11:21.349099 22248 catalog_manager.cc:1347] Generated new cluster ID: 32df7d01c9b447bea002322e6ab03f92
I20260927 21:11:21.349153 22248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:21.362824 22248 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:21.363008 22248 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:21.371598 22248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e7d2159706c42a6a8b2e4632490e9a9: Generated new TSK 0
I20260927 21:11:21.371830 22248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:21.378242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22155
2026-09-27T21:11:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:21 dist-test-slave-8bgs krb5kdc[22254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:21 dist-test-slave-8bgs krb5kdc[22254](info): set up 2 sockets
Sep 27 21:11:21 dist-test-slave-8bgs krb5kdc[22254](info): commencing operation
krb5kdc: starting...
W20260927 21:11:22.399168 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:22 dist-test-slave-8bgs krb5kdc[22254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:22.509696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:22.588366 22267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:22.588557 22267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:22.588578 22267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:22.589645 22267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:22.589680 22267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:22.589695 22267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:22.589707 22267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:22.589720 22267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:22.591076 22267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:22.591274 22267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:22.591432 22267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:22.596051 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:22.596060 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:22.596060 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:22.596529 22267 server_base.cc:1034] running on GCE node
I20260927 21:11:22.596709 22267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:22.596938 22267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:22.598076 22267 hybrid_clock.cc:648] HybridClock initialized: now 1790543482598060 us; error 29 us; skew 500 ppm
Sep 27 21:11:22 dist-test-slave-8bgs krb5kdc[22254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:22.600469 22267 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:22.600919 22267 webserver.cc:460] Webserver started at http://127.0.98.62:42961/ using document root <none> and password file <none>
I20260927 21:11:22.601142 22267 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:22.601186 22267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:22.601341 22267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:22.602317 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "36fa10b37e0148da92da8b0e856ea6bb"
format_stamp: "Formatted at 2026-09-27 21:11:22 on dist-test-slave-8bgs"
server_key: "a18a39b978b2ce33f5b82bf04cdee608"
server_key_iv: "b2e6b467032c883134cbd33124271d11"
server_key_version: "encryptionkey@0"
I20260927 21:11:22.602663 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "36fa10b37e0148da92da8b0e856ea6bb"
format_stamp: "Formatted at 2026-09-27 21:11:22 on dist-test-slave-8bgs"
server_key: "a18a39b978b2ce33f5b82bf04cdee608"
server_key_iv: "b2e6b467032c883134cbd33124271d11"
server_key_version: "encryptionkey@0"
I20260927 21:11:22.604022 22267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:11:22.604701 22282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:22.604889 22267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:22.604988 22267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "36fa10b37e0148da92da8b0e856ea6bb"
format_stamp: "Formatted at 2026-09-27 21:11:22 on dist-test-slave-8bgs"
server_key: "a18a39b978b2ce33f5b82bf04cdee608"
server_key_iv: "b2e6b467032c883134cbd33124271d11"
server_key_version: "encryptionkey@0"
I20260927 21:11:22.605058 22267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:22.629189 22267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:22.631109 22267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:22.631269 22267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:22.635311 22267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41723
I20260927 21:11:22.635340 22334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41723 every 8 connection(s)
I20260927 21:11:22.635686 22267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:22.636500 22335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:22.638697 22335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb: Bootstrap starting.
I20260927 21:11:22.639318 22335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:22.639595 22335 log.cc:824] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:22.640249 22335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb: No bootstrap required, opened a new log
I20260927 21:11:22.641805 22335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fa10b37e0148da92da8b0e856ea6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260927 21:11:22.641934 22335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:22.641956 22335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fa10b37e0148da92da8b0e856ea6bb, State: Initialized, Role: FOLLOWER
I20260927 21:11:22.642053 22335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fa10b37e0148da92da8b0e856ea6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260927 21:11:22.642135 22335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:22.642164 22335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:22.642233 22335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:22.642843 22335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fa10b37e0148da92da8b0e856ea6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260927 21:11:22.642956 22335 leader_election.cc:302] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36fa10b37e0148da92da8b0e856ea6bb; no voters:
I20260927 21:11:22.643150 22335 leader_election.cc:288] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:22.643299 22338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:22.643407 22335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:22.643473 22338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [term 1 LEADER]: Becoming Leader. State: Replica: 36fa10b37e0148da92da8b0e856ea6bb, State: Running, Role: LEADER
I20260927 21:11:22.643603 22338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fa10b37e0148da92da8b0e856ea6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260927 21:11:22.644104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22267
I20260927 21:11:22.644186 22338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36fa10b37e0148da92da8b0e856ea6bb. Latest consensus state: current_term: 1 leader_uuid: "36fa10b37e0148da92da8b0e856ea6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fa10b37e0148da92da8b0e856ea6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } } }
I20260927 21:11:22.644237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:22.644256 22338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:22.644400 392 external_mini_cluster.cc:1428] Setting key 8ba013935298e419df9201da66f4cc22
I20260927 21:11:22.644488 22347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:22.644884 22339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36fa10b37e0148da92da8b0e856ea6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fa10b37e0148da92da8b0e856ea6bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } } }
I20260927 21:11:22.645252 22339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:22.645542 22347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:22.646963 22347 catalog_manager.cc:1347] Generated new cluster ID: 6b05c01d9c1444aa9d315fc352627226
I20260927 21:11:22.647014 22347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:11:22 dist-test-slave-8bgs krb5kdc[22254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:22.654569 22347 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:22.655231 22347 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:22.656167 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:22.644989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46013 (local address 127.0.98.62:41723)
0927 21:11:22.645215 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:22.645224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:22.645310 (+ 86us) server_negotiation.cc:407] Connection header received
0927 21:11:22.645511 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:22.645518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:22.645581 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:22.645734 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:22.646474 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:22.647902 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:22.648986 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:22.649563 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:22.649654 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:22.652457 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:22.652462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:22.652463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:22.652476 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:22.654460 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:22.654857 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:22.654860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:22.654860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:22.654965 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:22.655224 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:22.655225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:22.655225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:22.655543 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:22.655708 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:22.655809 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:11:22.655956 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260927 21:11:22.657074 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:22.659888 22347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36fa10b37e0148da92da8b0e856ea6bb: Generated new TSK 0
I20260927 21:11:22.660044 22347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:22.665875 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:22.659351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35863 (local address 127.0.98.62:41723)
0927 21:11:22.659384 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:22.659386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:22.659706 (+ 320us) server_negotiation.cc:407] Connection header received
0927 21:11:22.659764 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:22.659765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:22.659777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:22.659834 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:22.660439 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:22.660918 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:22.661958 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:22.662140 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:22.662188 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:22.663507 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:22.663508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:22.663508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:22.663514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:22.664868 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:22.665236 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:22.665237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:22.665237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:22.665283 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:22.665471 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:22.665472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:22.665472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:22.665559 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:22.665626 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:22.665756 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:11:22.665822 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:22.668023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22267
2026-09-27T21:11:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:22 dist-test-slave-8bgs krb5kdc[22366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:22 dist-test-slave-8bgs krb5kdc[22366](info): set up 2 sockets
Sep 27 21:11:22 dist-test-slave-8bgs krb5kdc[22366](info): commencing operation
krb5kdc: starting...
W20260927 21:11:23.699244 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:23 dist-test-slave-8bgs krb5kdc[22366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:23.809863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34715
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:23.891206 22379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:23.891412 22379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:23.891436 22379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:23.892619 22379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:23.892661 22379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:23.892676 22379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:23.892714 22379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:23.892742 22379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:23.894286 22379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:23.894505 22379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:23.894707 22379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:23.899441 22385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:23.899482 22384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:23.899448 22387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:23.899509 22379 server_base.cc:1034] running on GCE node
I20260927 21:11:23.899893 22379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:23.900190 22379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:23.901360 22379 hybrid_clock.cc:648] HybridClock initialized: now 1790543483901347 us; error 28 us; skew 500 ppm
Sep 27 21:11:23 dist-test-slave-8bgs krb5kdc[22366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:23.903712 22379 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:23.904083 22379 webserver.cc:460] Webserver started at http://127.0.98.62:44895/ using document root <none> and password file <none>
I20260927 21:11:23.904269 22379 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:23.904307 22379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:23.904402 22379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:23.905376 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "358b66c379bc48d7ab9450055b2d0e2b"
format_stamp: "Formatted at 2026-09-27 21:11:23 on dist-test-slave-8bgs"
server_key: "11f3015c625ccbdb6cd618402663e07c"
server_key_iv: "b5c504754946d4403431ea59e3a4af18"
server_key_version: "encryptionkey@0"
I20260927 21:11:23.905663 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "358b66c379bc48d7ab9450055b2d0e2b"
format_stamp: "Formatted at 2026-09-27 21:11:23 on dist-test-slave-8bgs"
server_key: "11f3015c625ccbdb6cd618402663e07c"
server_key_iv: "b5c504754946d4403431ea59e3a4af18"
server_key_version: "encryptionkey@0"
I20260927 21:11:23.906855 22379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:23.907615 22394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:23.907765 22379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:23.907852 22379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "358b66c379bc48d7ab9450055b2d0e2b"
format_stamp: "Formatted at 2026-09-27 21:11:23 on dist-test-slave-8bgs"
server_key: "11f3015c625ccbdb6cd618402663e07c"
server_key_iv: "b5c504754946d4403431ea59e3a4af18"
server_key_version: "encryptionkey@0"
I20260927 21:11:23.907908 22379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:23.919938 22379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:23.922243 22379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:23.922359 22379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:23.926278 22379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34715
I20260927 21:11:23.926330 22446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34715 every 8 connection(s)
I20260927 21:11:23.926712 22379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:23.927686 22447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:23.929925 22447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b: Bootstrap starting.
I20260927 21:11:23.930487 22447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:23.930768 22447 log.cc:824] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:23.931525 22447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b: No bootstrap required, opened a new log
I20260927 21:11:23.933065 22447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358b66c379bc48d7ab9450055b2d0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34715 } }
I20260927 21:11:23.933224 22447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:23.933270 22447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358b66c379bc48d7ab9450055b2d0e2b, State: Initialized, Role: FOLLOWER
I20260927 21:11:23.933465 22447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358b66c379bc48d7ab9450055b2d0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34715 } }
I20260927 21:11:23.933566 22447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:23.933606 22447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:23.933657 22447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:23.934171 22447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358b66c379bc48d7ab9450055b2d0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34715 } }
I20260927 21:11:23.934188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22379
I20260927 21:11:23.934269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:23.934288 22447 leader_election.cc:302] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 358b66c379bc48d7ab9450055b2d0e2b; no voters:
I20260927 21:11:23.934485 392 external_mini_cluster.cc:1428] Setting key 3bd92b764876e1f146fc326a0c49ca56
I20260927 21:11:23.934490 22447 leader_election.cc:288] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:23.934520 22450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:23.934631 22450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [term 1 LEADER]: Becoming Leader. State: Replica: 358b66c379bc48d7ab9450055b2d0e2b, State: Running, Role: LEADER
I20260927 21:11:23.934748 22447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:23.934767 22450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358b66c379bc48d7ab9450055b2d0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34715 } }
I20260927 21:11:23.935297 22452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 358b66c379bc48d7ab9450055b2d0e2b. Latest consensus state: current_term: 1 leader_uuid: "358b66c379bc48d7ab9450055b2d0e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358b66c379bc48d7ab9450055b2d0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34715 } } }
I20260927 21:11:23.935269 22451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "358b66c379bc48d7ab9450055b2d0e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "358b66c379bc48d7ab9450055b2d0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34715 } } }
I20260927 21:11:23.935457 22451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:23.935443 22452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:23.936607 22468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:23.936666 22468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:23.936708 22461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:23.936857 22461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:23.938416 22461 catalog_manager.cc:1347] Generated new cluster ID: 7b79042ca9d34fe78700aeb8d912bf7c
I20260927 21:11:23.938468 22461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:23.940384 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:23.935263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48693 (local address 127.0.98.62:34715)
0927 21:11:23.935620 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:23.935630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:23.935667 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:11:23.935805 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:23.935813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:23.935889 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:23.936046 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:23.936329 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:23.937324 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:23.938399 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:23.938902 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:23.939021 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:23.939737 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:23.939741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:23.939741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:23.939929 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:23.939957 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:23.939968 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:23.940043 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:23.940118 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:11:23.940179 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":67,"threads_started":1}
I20260927 21:11:23.943506 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:23.947665 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:23.944843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60381 (local address 127.0.98.62:34715)
0927 21:11:23.944878 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:23.944880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:23.944996 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:11:23.945066 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:23.945067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:23.945091 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:23.945146 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:23.945277 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:23.945735 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:23.946435 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:23.946660 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:23.946743 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:23.947428 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:23.947430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:23.947430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:23.947434 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:23.947447 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:23.947451 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:23.947503 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:23.947556 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:11:23.947577 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:11:23.958262 22461 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:23.958794 22461 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:23.976501 22461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 358b66c379bc48d7ab9450055b2d0e2b: Generated new TSK 0
I20260927 21:11:23.976706 22461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:23.984561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22379
2026-09-27T21:11:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:24 dist-test-slave-8bgs krb5kdc[22478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:24 dist-test-slave-8bgs krb5kdc[22478](info): set up 2 sockets
Sep 27 21:11:24 dist-test-slave-8bgs krb5kdc[22478](info): commencing operation
krb5kdc: starting...
W20260927 21:11:25.007282 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:25 dist-test-slave-8bgs krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:25.122066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:25.201118 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:25.201361 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:25.201398 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:25.202538 22491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:25.202610 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:25.202633 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:25.202654 22491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:25.202682 22491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:25.204177 22491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:25.204453 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:25.204681 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:25.209388 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:25.209609 22491 server_base.cc:1034] running on GCE node
W20260927 21:11:25.209419 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:25.209447 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:25.209905 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:25.210239 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:25.211405 22491 hybrid_clock.cc:648] HybridClock initialized: now 1790543485211399 us; error 35 us; skew 500 ppm
Sep 27 21:11:25 dist-test-slave-8bgs krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:25.214074 22491 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:25.214468 22491 webserver.cc:460] Webserver started at http://127.0.98.62:46209/ using document root <none> and password file <none>
I20260927 21:11:25.214648 22491 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:25.214687 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:25.214774 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:25.215837 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "78bc6e903a6e4051ae2ff52695eb7200"
format_stamp: "Formatted at 2026-09-27 21:11:25 on dist-test-slave-8bgs"
server_key: "df42803f9946d0b187e0935e72d66676"
server_key_iv: "7d11f5383f83e5622098f6920adad99f"
server_key_version: "encryptionkey@0"
I20260927 21:11:25.216141 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "78bc6e903a6e4051ae2ff52695eb7200"
format_stamp: "Formatted at 2026-09-27 21:11:25 on dist-test-slave-8bgs"
server_key: "df42803f9946d0b187e0935e72d66676"
server_key_iv: "7d11f5383f83e5622098f6920adad99f"
server_key_version: "encryptionkey@0"
I20260927 21:11:25.217391 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:25.218060 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:25.218230 22491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:25.218335 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "78bc6e903a6e4051ae2ff52695eb7200"
format_stamp: "Formatted at 2026-09-27 21:11:25 on dist-test-slave-8bgs"
server_key: "df42803f9946d0b187e0935e72d66676"
server_key_iv: "7d11f5383f83e5622098f6920adad99f"
server_key_version: "encryptionkey@0"
I20260927 21:11:25.218410 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:25.221800 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:25.224093 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:25.224254 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:25.228068 22491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41503
I20260927 21:11:25.228096 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41503 every 8 connection(s)
I20260927 21:11:25.228412 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:25.229307 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:25.231644 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200: Bootstrap starting.
I20260927 21:11:25.232198 22559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:25.232481 22559 log.cc:824] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:25.233130 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200: No bootstrap required, opened a new log
I20260927 21:11:25.234692 22559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc6e903a6e4051ae2ff52695eb7200" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41503 } }
I20260927 21:11:25.234825 22559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:11:25.234848 22559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bc6e903a6e4051ae2ff52695eb7200, State: Initialized, Role: FOLLOWER
I20260927 21:11:25.234948 22559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc6e903a6e4051ae2ff52695eb7200" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41503 } }
I20260927 21:11:25.235018 22559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:11:25.235054 22559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:11:25.235092 22559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:25.235653 22559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc6e903a6e4051ae2ff52695eb7200" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41503 } }
I20260927 21:11:25.235762 22559 leader_election.cc:302] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bc6e903a6e4051ae2ff52695eb7200; no voters:
I20260927 21:11:25.235935 22559 leader_election.cc:288] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:11:25.235999 22562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:25.236145 22562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [term 1 LEADER]: Becoming Leader. State: Replica: 78bc6e903a6e4051ae2ff52695eb7200, State: Running, Role: LEADER
I20260927 21:11:25.236182 22559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:25.236251 22562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc6e903a6e4051ae2ff52695eb7200" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41503 } }
I20260927 21:11:25.236690 22564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bc6e903a6e4051ae2ff52695eb7200. Latest consensus state: current_term: 1 leader_uuid: "78bc6e903a6e4051ae2ff52695eb7200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc6e903a6e4051ae2ff52695eb7200" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41503 } } }
I20260927 21:11:25.236697 22563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bc6e903a6e4051ae2ff52695eb7200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bc6e903a6e4051ae2ff52695eb7200" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41503 } } }
I20260927 21:11:25.236778 22564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:25.236774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22491
I20260927 21:11:25.236806 22563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:25.236896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:25.237015 22566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:25.237099 392 external_mini_cluster.cc:1428] Setting key f568aa15b36cfa9badcab97458fc4c5c
I20260927 21:11:25.237601 22566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:25.239341 22566 catalog_manager.cc:1347] Generated new cluster ID: 1affc7d8ab104571a7d78604f426b2d8
I20260927 21:11:25.239396 22566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:25.239734 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:25.237703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34215 (local address 127.0.98.62:41503)
0927 21:11:25.237866 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:25.237884 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:25.237914 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:11:25.238239 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:25.238247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:25.238281 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:25.238423 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:25.239156 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:25.239160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:25.239160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:25.239164 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:25.239211 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:25.239222 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:25.239553 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:25.239578 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 21:11:25.239610 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260927 21:11:25.240504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:25.244179 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:25.241827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58469 (local address 127.0.98.62:41503)
0927 21:11:25.241878 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:25.241879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:25.243245 (+ 1366us) server_negotiation.cc:407] Connection header received
0927 21:11:25.243326 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:25.243326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:25.243331 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:25.243391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:25.243955 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:25.243956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:25.243956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:11:25.243960 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:25.243973 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:11:25.243977 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:25.244032 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:25.244087 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 21:11:25.244101 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:11:25.255462 22566 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:25.255625 22566 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:25.261011 22566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bc6e903a6e4051ae2ff52695eb7200: Generated new TSK 0
I20260927 21:11:25.261190 22566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:11:25.266726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22491
2026-09-27T21:11:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1281 ms)
[----------] 18 tests from AuthTokenIssuingTest (23532 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:25 dist-test-slave-8bgs krb5kdc[22590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:25 dist-test-slave-8bgs krb5kdc[22590](info): set up 2 sockets
Sep 27 21:11:25 dist-test-slave-8bgs krb5kdc[22590](info): commencing operation
krb5kdc: starting...
W20260927 21:11:26.299381 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:26.413247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43469
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43469,127.0.98.61:41423,127.0.98.60:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:26.492928 22603 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:11:26.493141 22603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:26.493158 22603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:26.493207 22603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:26.494354 22603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:26.494395 22603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:26.494410 22603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:26.494458 22603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:26.494477 22603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:26.496097 22603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43469,127.0.98.61:41423,127.0.98.60:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:26.496353 22603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:26.496587 22603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:26.501358 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:26.501355 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:26.501441 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:26.501406 22603 server_base.cc:1034] running on GCE node
I20260927 21:11:26.501802 22603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:26.502107 22603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:26.503271 22603 hybrid_clock.cc:648] HybridClock initialized: now 1790543486503265 us; error 35 us; skew 500 ppm
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:26.509465 22603 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:26.509938 22603 webserver.cc:460] Webserver started at http://127.0.98.62:41383/ using document root <none> and password file <none>
I20260927 21:11:26.510183 22603 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:26.510251 22603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:26.510407 22603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:26.511567 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "bed5791eef164ecaaaa9b310e1b233ac"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "7278e74ffecec08fb1ec314596e3483b"
server_key_iv: "fad9f4e002a88fcfee127c5c00d66a2a"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.511929 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "bed5791eef164ecaaaa9b310e1b233ac"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "7278e74ffecec08fb1ec314596e3483b"
server_key_iv: "fad9f4e002a88fcfee127c5c00d66a2a"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.513217 22603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:26.513933 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:26.514117 22603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:11:26.514216 22603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "bed5791eef164ecaaaa9b310e1b233ac"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "7278e74ffecec08fb1ec314596e3483b"
server_key_iv: "fad9f4e002a88fcfee127c5c00d66a2a"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.514297 22603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:26.522260 22603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:26.524099 22603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:26.524328 22603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:26.528386 22603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43469
I20260927 21:11:26.528439 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43469 every 8 connection(s)
I20260927 21:11:26.528780 22603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:26.529685 22671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:26.531214 22671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:26.537510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22603
I20260927 21:11:26.537622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:26.537829 392 external_mini_cluster.cc:1428] Setting key 5852cd65d4e4eaa59bc61b6fbcc96211
I20260927 21:11:26.540624 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.531847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43469 (local address 127.0.0.1:56370)
0927 21:11:26.532084 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.532111 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.532551 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.532800 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.532802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.533035 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.533184 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.533187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.534150 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.534151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.534823 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.534825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.535350 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.535351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.535402 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.535906 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.535914 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.537198 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.538964 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.538972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.538974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.539284 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.539553 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.539554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.539554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.539633 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.540049 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.540053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.540106 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.540175 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.540321 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260927 21:11:26.540624 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.531869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.98.62:43469)
0927 21:11:26.532028 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.532036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.532192 (+ 156us) server_negotiation.cc:407] Connection header received
0927 21:11:26.532675 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.532677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.532693 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.532771 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.533267 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.534067 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.534899 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.535263 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.535344 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.537270 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.537276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.537276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.537281 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.538881 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.539363 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.539366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.539366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.539482 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.539697 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.539697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.539698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.539977 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.540072 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.540232 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.540395 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260927 21:11:26.541420 22671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:26.541774 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.541659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36604)
0927 21:11:26.541705 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.541746 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:11:26.541934 22621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
I20260927 21:11:26.542339 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.542253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36618)
0927 21:11:26.542299 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.542323 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:11:26.542472 22671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 21:11:26.559913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41423
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43469,127.0.98.61:41423,127.0.98.60:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:11:26.591677 22671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attempt: 1
I20260927 21:11:26.592175 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.592050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36630)
0927 21:11:26.592107 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.592146 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:11:26.592505 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.592414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36634)
0927 21:11:26.592460 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.592482 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:11:26.592585 22671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
W20260927 21:11:26.638585 22678 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:11:26.638824 22678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:26.638854 22678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:26.638897 22678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:26.640009 22678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:26.640076 22678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:26.640100 22678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:26.640120 22678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:26.640149 22678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:26.641582 22678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43469,127.0.98.61:41423,127.0.98.60:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:26.641867 22678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:26.642084 22678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:26.646690 22684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:26.646699 22686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:26.646937 22678 server_base.cc:1034] running on GCE node
W20260927 21:11:26.646819 22683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:26.647264 22678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:26.647552 22678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:26.648692 22678 hybrid_clock.cc:648] HybridClock initialized: now 1790543486648668 us; error 45 us; skew 500 ppm
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:26.651010 22678 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 21:11:26.651443 22678 webserver.cc:460] Webserver started at http://127.0.98.61:37555/ using document root <none> and password file <none>
I20260927 21:11:26.651660 22678 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:26.651703 22678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:26.651866 22678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:26.652858 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data/instance:
uuid: "6118a617dfeb4d21a9377e607b77c1f4"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "7a554129c82e426c419e77d1c8bb36b8"
server_key_iv: "2870f5e838ac92de30d15f3a75f23034"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.653190 22678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal/instance:
uuid: "6118a617dfeb4d21a9377e607b77c1f4"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "7a554129c82e426c419e77d1c8bb36b8"
server_key_iv: "2870f5e838ac92de30d15f3a75f23034"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.654455 22678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:11:26.655143 22693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:26.655344 22678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:26.655439 22678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal
uuid: "6118a617dfeb4d21a9377e607b77c1f4"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "7a554129c82e426c419e77d1c8bb36b8"
server_key_iv: "2870f5e838ac92de30d15f3a75f23034"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.655526 22678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:26.660712 22671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attempt: 2
I20260927 21:11:26.661069 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.660955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36642)
0927 21:11:26.661008 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.661043 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 21:11:26.661381 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.661295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36658)
0927 21:11:26.661335 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.661358 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 21:11:26.661464 22671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41423: connect: Connection refused (error 111)
I20260927 21:11:26.677706 22678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:26.679473 22678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:26.679625 22678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:26.683372 22678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41423
I20260927 21:11:26.683418 22745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41423 every 8 connection(s)
I20260927 21:11:26.683776 22678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:11:26.684043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22678
I20260927 21:11:26.684159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-1/wal/instance
I20260927 21:11:26.684367 392 external_mini_cluster.cc:1428] Setting key 507f6b03e20468466bb45dfbe2911c92
I20260927 21:11:26.684883 22746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:26.686896 22746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:26.697851 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.687609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.0.98.62:43469)
0927 21:11:26.687675 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.687677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.687846 (+ 169us) server_negotiation.cc:407] Connection header received
0927 21:11:26.688289 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.688291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.688306 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.688381 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.688997 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.689477 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.690418 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.690625 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.690707 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.695129 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.695131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.695131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.695140 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.696502 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.696909 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.696910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.696910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.696975 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.697209 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.697210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.697210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.697328 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.697400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.697674 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.697757 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:11:26.698103 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.687544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43469 (local address 127.0.0.1:56374)
0927 21:11:26.687721 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.687760 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.688195 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.688487 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.688490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.688719 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.688901 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.688905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.689578 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.689579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.690331 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.690333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.690732 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.690733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.690857 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.691396 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.691403 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.695007 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.696599 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.696606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.696608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.696821 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.697037 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.697038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.697038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.697144 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.697392 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.697398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.697542 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.697640 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.697815 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260927 21:11:26.698809 22746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:11:26.707046 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.699111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36664)
0927 21:11:26.699245 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.699253 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.699289 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.699478 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.699479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.699583 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.699632 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.699633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.700919 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.700919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.701389 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.701390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.702035 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.702035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.702085 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.702474 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.702480 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.704350 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.705921 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.705922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.705922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.706136 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.706382 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.706383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.706383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.706484 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.706767 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.706768 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.706819 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.706899 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.706983 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 21:11:26.707231 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.699143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.0.98.61:41423)
0927 21:11:26.699303 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.699309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.699336 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:11:26.699361 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.699363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.699380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.699463 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.699701 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.700846 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.701466 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.701957 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.702041 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.704425 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.704432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.704432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.704438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.705829 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.706216 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.706219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.706219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.706269 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.706555 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.706556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.706556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.706680 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.706757 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.706970 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.707036 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 21:11:26.707765 22746 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:26.708084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36733
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43469,127.0.98.61:41423,127.0.98.60:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:11:26.708132 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.707996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60440)
0927 21:11:26.708051 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.708107 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:11:26.708294 22697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
I20260927 21:11:26.708745 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.708640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60442)
0927 21:11:26.708685 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.708714 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:11:26.708912 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
I20260927 21:11:26.752595 22671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } attempt: 3
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:11:26.758131 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } attempt: 1
I20260927 21:11:26.758548 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.758409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60450)
0927 21:11:26.758472 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.758506 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:11:26.758963 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.758847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60452)
0927 21:11:26.758898 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.758930 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:11:26.759119 22746 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
I20260927 21:11:26.760082 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.753000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36668)
0927 21:11:26.753082 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.753095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.753160 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.753299 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.753299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.753447 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.753513 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.753515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.754453 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.754454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.754980 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.754982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.755499 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.755500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.755571 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.756040 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.756045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.757196 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.759068 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.759069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.759070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.759289 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.759474 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.759475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.759475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.759532 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.759780 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.759781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.759828 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.759894 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.759989 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 21:11:26.760082 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.753066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.0.98.61:41423)
0927 21:11:26.753145 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.753148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.753160 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:11:26.753222 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.753223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.753234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.753286 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.753587 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.754368 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.755078 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.755417 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.755501 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.757272 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.757273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.757273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.757281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.758972 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.759358 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.759359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.759359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.759402 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.759597 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.759598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.759598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.759707 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.759776 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.759931 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.760007 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:11:26.760463 22671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:26.760771 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.760704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60464)
0927 21:11:26.760731 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.760756 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 21:11:26.761106 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.761022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60476)
0927 21:11:26.761062 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.761087 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:11:26.761190 22671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
W20260927 21:11:26.788786 22753 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:11:26.789017 22753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:26.789047 22753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:26.789088 22753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:26.790175 22753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:26.790239 22753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:26.790263 22753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:26.790292 22753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:26.790323 22753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:26.791757 22753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43469,127.0.98.61:41423,127.0.98.60:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:26.792029 22753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:26.792249 22753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:11:26.792346 22671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } attempt: 1
I20260927 21:11:26.792766 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.792641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60486)
0927 21:11:26.792705 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.792740 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:11:26.793071 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.792977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60500)
0927 21:11:26.793022 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.793047 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:11:26.793151 22671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
W20260927 21:11:26.797120 22759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:26.797132 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:26.797173 22753 server_base.cc:1034] running on GCE node
W20260927 21:11:26.797171 22758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:26.797634 22753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:26.797858 22753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:26.798992 22753 hybrid_clock.cc:648] HybridClock initialized: now 1790543486798979 us; error 25 us; skew 500 ppm
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:26.801353 22753 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 21:11:26.801803 22753 webserver.cc:460] Webserver started at http://127.0.98.60:44005/ using document root <none> and password file <none>
I20260927 21:11:26.802022 22753 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:26.802065 22753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:26.802206 22753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:26.803292 22753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data/instance:
uuid: "8b9f310c2967404193ba63a2c9b2eb8b"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "3de9dfe1443d1ceaa299ded066e59ba6"
server_key_iv: "4f8945bef151f630010d7c1728075ff7"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.803638 22753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal/instance:
uuid: "8b9f310c2967404193ba63a2c9b2eb8b"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "3de9dfe1443d1ceaa299ded066e59ba6"
server_key_iv: "4f8945bef151f630010d7c1728075ff7"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.804924 22753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:11:26.805627 22768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:26.805836 22753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:26.805924 22753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal
uuid: "8b9f310c2967404193ba63a2c9b2eb8b"
format_stamp: "Formatted at 2026-09-27 21:11:26 on dist-test-slave-8bgs"
server_key: "3de9dfe1443d1ceaa299ded066e59ba6"
server_key_iv: "4f8945bef151f630010d7c1728075ff7"
server_key_version: "encryptionkey@0"
I20260927 21:11:26.806005 22753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:26.821846 22753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:26.823763 22753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:26.823916 22753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:26.827298 22746 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } attempt: 2
I20260927 21:11:26.827708 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.827577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60514)
0927 21:11:26.827640 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.827677 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:11:26.828066 22753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36733
I20260927 21:11:26.828101 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36733 every 8 connection(s)
I20260927 21:11:26.828449 22753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:11:26.829169 22821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:26.831151 22821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:26.832399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22753
I20260927 21:11:26.832497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790543220945705-392-0/minicluster-data/master-2/wal/instance
I20260927 21:11:26.832731 392 external_mini_cluster.cc:1428] Setting key 17c3f5cb6e1736c088b3f4fa4ccfb18c
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:26.838655 22749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.828036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60526)
0927 21:11:26.828086 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.828097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.828141 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.829732 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.829733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.829860 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.829902 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.829904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.831531 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.831532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.832297 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.832299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.832996 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.832997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.833065 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.833518 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.833524 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.835028 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.837273 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.837274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.837274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.837512 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.837772 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.837772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.837773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.837829 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.838305 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.838306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.838356 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.838450 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.838554 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:11:26.838841 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.828330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.98.60:36733)
0927 21:11:26.828885 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.828894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.828926 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:11:26.829512 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.829521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.829579 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.829725 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.830005 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.831377 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.832371 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.832917 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.833032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.835153 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.835160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.835160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.835172 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.837193 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.837594 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.837597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.837598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.837690 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.837909 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.837910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.837910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.838191 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.838274 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.838521 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.838637 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":493,"threads_started":1}
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:11:26.840400 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.831866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.0.98.62:43469)
0927 21:11:26.831909 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.831911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.832078 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:11:26.832108 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.832109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.832119 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.832181 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.832787 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.833750 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.834771 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.834994 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.835101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.837263 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.837264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.837265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.837273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.839013 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.839483 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.839485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.839485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.839544 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.839729 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.839730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.839731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.839868 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.839933 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.840238 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.840307 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:11:26.840502 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.831796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43469 (local address 127.0.0.1:56376)
0927 21:11:26.831979 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.832009 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.832066 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.832195 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.832197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.832604 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.832699 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.832702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.833836 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.833837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.834671 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.834674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.835133 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.835135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.835216 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.835660 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.835664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.837171 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.839128 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.839132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.839134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.839401 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.839613 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.839614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.839614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.839671 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.839926 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.839931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.839981 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.840086 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.840217 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260927 21:11:26.840797 22746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: Bootstrap starting.
I20260927 21:11:26.841568 22746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:26.841864 22746 log.cc:824] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:26.842427 22821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:26.842669 22746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: No bootstrap required, opened a new log
I20260927 21:11:26.842725 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.833257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.0.98.60:36733)
0927 21:11:26.833395 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.833397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.833404 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:11:26.833421 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.833422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.833430 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.833527 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.834296 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.835037 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.836017 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.836318 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.836397 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.839669 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.839670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.839670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.839676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.841509 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.841890 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.841891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.841891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.841973 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.842184 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.842185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.842185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.842320 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.842451 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.842537 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.842612 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
I20260927 21:11:26.844681 22746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.844806 22746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6118a617dfeb4d21a9377e607b77c1f4, State: Initialized, Role: FOLLOWER
I20260927 21:11:26.844890 22746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.845175 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.845216 22746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:26.845270 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260927 21:11:26.847070 22839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:26.847242 22839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:26.847371 22678 master_runner.cc:186] Error getting master registration for 127.0.98.62:43469: OK, instance_id { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" instance_seqno: 1790543486527282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:11:26.850116 22678 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:11:26.850852 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.843555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.0.98.62:43469)
0927 21:11:26.843587 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.843589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.843629 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:11:26.843695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.843696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.843705 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.843758 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.844185 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.844629 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.845491 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.845746 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.845808 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.848199 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.848200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.848201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.848208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.849585 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.849930 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.849931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.849931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.849995 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.850197 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.850197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.850198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.850308 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.850422 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.850670 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.850754 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 21:11:26.853387 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.842650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41423 (local address 127.0.0.1:36682)
0927 21:11:26.842724 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.842734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.842766 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.842974 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.842975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.843142 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.843187 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.843189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.844052 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.844053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.844803 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.844805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.845450 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.845451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.845533 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.845916 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.845922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.847424 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.852328 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.852329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.852329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.852546 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.852751 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.852752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.852752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.852804 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.853088 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.853089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.853131 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.853205 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.853293 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 21:11:26.853417 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.842841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.0.98.61:41423)
0927 21:11:26.842878 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.842880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.842886 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:11:26.842903 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.842904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.842913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.842968 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.843279 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.843956 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.844932 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.845199 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.845263 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.850193 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.850194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.850194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.850201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.852233 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.852629 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.852631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.852631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.852684 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.852867 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.852868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.852868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.853010 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.853080 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.853258 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.853330 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:11:26.853971 22821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:11:26.859910 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.851452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.0.98.61:41423)
0927 21:11:26.851579 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.851581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.851697 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:11:26.851766 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.851767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.851777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.851844 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.852282 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.852797 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.853705 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.854025 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.854319 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.856456 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.856457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.856458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.856465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.858224 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.858793 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.858794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.858794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.858853 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.859067 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.859068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.859068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.859216 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.859672 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.859713 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.859788 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260927 21:11:26.861351 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.854246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60532)
0927 21:11:26.854333 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.854341 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.854385 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.854523 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.854524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.854675 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.854711 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.854712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.855619 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.855620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.856357 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.856360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.856920 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.856922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.856985 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.857370 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.857374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.858694 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.860333 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.860334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.860334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.860551 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.860736 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.860736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.860737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.860780 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.861084 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.861085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.861120 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.861178 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.861259 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:11:26.861572 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:26.861608 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.854281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.98.60:36733)
0927 21:11:26.854314 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.854316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.854385 (+ 69us) server_negotiation.cc:407] Connection header received
0927 21:11:26.854441 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.854442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.854460 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.854515 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.854795 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.855531 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.856465 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.856857 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.856917 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.858783 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.858784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.858784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.858790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.860255 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.860634 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.860634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.860635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.860680 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.860834 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.860834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.860835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.861001 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.861063 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.861474 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.861538 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:26.863706 22821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: Bootstrap starting.
I20260927 21:11:26.862735 22645 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bed5791eef164ecaaaa9b310e1b233ac"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56392
I20260927 21:11:26.864437 22821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:26.864774 22821 log.cc:824] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:26.865526 22821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: No bootstrap required, opened a new log
I20260927 21:11:26.866932 22821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.867089 22821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9f310c2967404193ba63a2c9b2eb8b, State: Initialized, Role: FOLLOWER
I20260927 21:11:26.867218 22821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.867564 22821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:26.867520 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.867710 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:11:26.869866 22854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:26.869925 22854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:26.871155 22671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } attempt: 2
I20260927 21:11:26.871707 22753 master_runner.cc:186] Error getting master registration for 127.0.98.62:43469: OK, instance_id { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" instance_seqno: 1790543486527282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:11:26.873984 22753 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 21:11:26 dist-test-slave-8bgs krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:11:26.878731 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.871509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36733 (local address 127.0.0.1:60548)
0927 21:11:26.871590 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:26.871600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:26.871638 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:26.871893 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:26.871894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:26.872074 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:26.872121 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.872122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.872746 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.872747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.873545 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:26.873546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.873993 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.873994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:26.874048 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.874500 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:26.874504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:26.875883 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:26.877703 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.877704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.877705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.877880 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.878079 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:26.878094 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:26.878095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:26.878149 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:26.878415 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:26.878416 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:26.878477 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 21:11:26.878540 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:11:26.878633 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:11:26.878731 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.871717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.0.98.60:36733)
0927 21:11:26.871767 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.871769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.871777 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:11:26.871797 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.871798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.871806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.871889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.872188 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.872662 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.873621 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.873933 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.873992 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.875961 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.875963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.875963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.875969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.877635 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.877965 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.877967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.877967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.878021 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.878214 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.878215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.878215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.878358 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.878419 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.878571 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.878650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:26.880244 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac: Bootstrap starting.
I20260927 21:11:26.880936 22671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:26.881242 22671 log.cc:824] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:26.882009 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac: No bootstrap required, opened a new log
I20260927 21:11:26.883622 22671 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.883780 22671 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed5791eef164ecaaaa9b310e1b233ac, State: Initialized, Role: FOLLOWER
I20260927 21:11:26.883867 22671 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.884186 22671 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:26.884239 22645 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:11:26.884305 22645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:26.885123 22645 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.885403 22645 leader_election.cc:288] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [CANDIDATE]: Term 1 election: Requested vote from peers 6118a617dfeb4d21a9377e607b77c1f4 (127.0.98.61:41423), 8b9f310c2967404193ba63a2c9b2eb8b (127.0.98.60:36733)
I20260927 21:11:26.884150 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.885624 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:26.885699 22720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bed5791eef164ecaaaa9b310e1b233ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6118a617dfeb4d21a9377e607b77c1f4"
I20260927 21:11:26.885898 22720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:26.886569 22720 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed5791eef164ecaaaa9b310e1b233ac in term 1.
I20260927 21:11:26.886825 22621 leader_election.cc:302] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6118a617dfeb4d21a9377e607b77c1f4, bed5791eef164ecaaaa9b310e1b233ac; no voters:
I20260927 21:11:26.887984 22795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bed5791eef164ecaaaa9b310e1b233ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b9f310c2967404193ba63a2c9b2eb8b"
I20260927 21:11:26.888127 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:26.889103 22795 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bed5791eef164ecaaaa9b310e1b233ac in term 1.
I20260927 21:11:26.889770 22857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:26.890264 22857 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [term 1 LEADER]: Becoming Leader. State: Replica: bed5791eef164ecaaaa9b310e1b233ac, State: Running, Role: LEADER
I20260927 21:11:26.890408 22857 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } }
I20260927 21:11:26.891573 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed5791eef164ecaaaa9b310e1b233ac. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.891662 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: This master's current role is: LEADER
W20260927 21:11:26.892556 22868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:26.892619 22868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:26.897362 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.887999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.0.98.62:43469)
0927 21:11:26.888040 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.888042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.888484 (+ 442us) server_negotiation.cc:407] Connection header received
0927 21:11:26.888592 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.888592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.888602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.888633 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.889571 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.890265 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.891315 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.891594 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.891635 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.894356 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.894357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.894358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.894365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.896168 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.896544 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.896545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.896546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.896594 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.896793 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.896794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.896794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.896911 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.896995 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.897187 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.897273 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:11:26.898195 22878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:26.898386 22878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:26.899762 22795 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [term 1 FOLLOWER]: Refusing update from remote peer bed5791eef164ecaaaa9b310e1b233ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:26.900089 22857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:26.901587 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed5791eef164ecaaaa9b310e1b233ac. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.901695 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:26.902130 22720 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [term 1 FOLLOWER]: Refusing update from remote peer bed5791eef164ecaaaa9b310e1b233ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:26.902441 22876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:26.903865 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed5791eef164ecaaaa9b310e1b233ac. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.903967 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:26.904907 22876 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.905004 22876 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:26.905368 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.905454 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:26.905694 22880 mvcc.cc:205] Tried to move back new op lower bound from 7334066122339590144 to 7334066122304008192. Current Snapshot: MvccSnapshot[applied={T|T < 7334066122339590144}]
I20260927 21:11:26.906248 22883 mvcc.cc:205] Tried to move back new op lower bound from 7334066122339590144 to 7334066122304008192. Current Snapshot: MvccSnapshot[applied={T|T < 7334066122339590144}]
I20260927 21:11:26.907701 22878 catalog_manager.cc:1347] Generated new cluster ID: 9f612c5057ce45b09d04fad4dba7dee3
I20260927 21:11:26.907745 22878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:26.912127 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.889371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.0.98.61:41423)
0927 21:11:26.889435 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.889437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.893316 (+ 3879us) server_negotiation.cc:407] Connection header received
0927 21:11:26.893432 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.893433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.893445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.893475 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.895907 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.896641 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.900036 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.900353 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.900398 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.903986 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.903987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.903988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.903994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.905702 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.911192 (+ 5490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.911193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.911194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.911248 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.911474 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.911475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.911475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.911623 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.911725 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.911928 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.912030 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:11:26.912261 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:26.890017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.0.98.60:36733)
0927 21:11:26.890049 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.890050 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.895108 (+ 5058us) server_negotiation.cc:407] Connection header received
0927 21:11:26.895206 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.895207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.895217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.895248 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.898108 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.898755 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.902073 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:26.902350 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:26.902391 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:26.906094 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:26.906095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:26.906095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:26.906102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:26.907614 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.911468 (+ 3854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.911470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.911470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.911521 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:26.911711 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:26.911711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:26.911712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:26.911843 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:26.911920 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:26.912100 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:11:26.912169 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:26.917182 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.917274 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:26.917726 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bed5791eef164ecaaaa9b310e1b233ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bed5791eef164ecaaaa9b310e1b233ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43469 } } peers { permanent_uuid: "6118a617dfeb4d21a9377e607b77c1f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41423 } } peers { permanent_uuid: "8b9f310c2967404193ba63a2c9b2eb8b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36733 } } }
I20260927 21:11:26.917810 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:26.918045 22881 mvcc.cc:205] Tried to move back new op lower bound from 7334066122339590144 to 7334066122304008192. Current Snapshot: MvccSnapshot[applied={T|T < 7334066122339590144}]
I20260927 21:11:26.935777 22878 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:26.936510 22878 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:26.941174 22878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed5791eef164ecaaaa9b310e1b233ac: Generated new TSK 0
I20260927 21:11:26.941395 22878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:11:26.961048 22826 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.959100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.98.60:36733)
0927 21:11:26.959143 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.959145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.960101 (+ 956us) server_negotiation.cc:407] Connection header received
0927 21:11:26.960349 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.960350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.960361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.960394 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.960979 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60570: BlockingRecv error: recv got EOF from 127.0.0.1:60570 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:11:26.961540 22674 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.959499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.0.98.62:43469)
0927 21:11:26.959538 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.959540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.960187 (+ 647us) server_negotiation.cc:407] Connection header received
0927 21:11:26.960459 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.960460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.960470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.960498 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.961479 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56410: BlockingRecv error: recv got EOF from 127.0.0.1:56410 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:11:26.962314 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.959792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.0.98.61:41423)
0927 21:11:26.959831 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.959833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.961748 (+ 1915us) server_negotiation.cc:407] Connection header received
0927 21:11:26.961927 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.961928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.961939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.961968 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.962258 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36712: BlockingRecv error: recv got EOF from 127.0.0.1:36712 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:11:26.964509 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.963815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.0.98.61:41423)
0927 21:11:26.963846 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.963848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.964033 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:11:26.964130 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.964131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.964147 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.964175 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.964464 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36722: BlockingRecv error: recv got EOF from 127.0.0.1:36722 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 21:11:26.967391 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.966593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.0.98.61:41423)
0927 21:11:26.966623 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.966625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.966879 (+ 254us) server_negotiation.cc:407] Connection header received
0927 21:11:26.966970 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.966970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.966979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.967007 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.967350 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36738: BlockingRecv error: recv got EOF from 127.0.0.1:36738 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 21:11:26.971067 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.970347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.98.61:41423)
0927 21:11:26.970378 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.970379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.970595 (+ 216us) server_negotiation.cc:407] Connection header received
0927 21:11:26.970683 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.970684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.970694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.970723 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.971024 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36748: BlockingRecv error: recv got EOF from 127.0.0.1:36748 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 21:11:26.976723 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.976026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.0.98.61:41423)
0927 21:11:26.976059 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.976060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.976272 (+ 212us) server_negotiation.cc:407] Connection header received
0927 21:11:26.976365 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.976366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.976375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.976404 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.976682 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36752: BlockingRecv error: recv got EOF from 127.0.0.1:36752 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 21:11:26.986543 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:26.985802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.98.61:41423)
0927 21:11:26.985843 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:26.985845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:26.986073 (+ 228us) server_negotiation.cc:407] Connection header received
0927 21:11:26.986164 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:26.986165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:26.986174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:26.986202 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:26.986499 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36764: BlockingRecv error: recv got EOF from 127.0.0.1:36764 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 21:11:27.004613 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:27.003720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.0.98.61:41423)
0927 21:11:27.003771 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:27.003773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:27.004048 (+ 275us) server_negotiation.cc:407] Connection header received
0927 21:11:27.004146 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:27.004147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:27.004158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:27.004188 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:27.004557 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36772: BlockingRecv error: recv got EOF from 127.0.0.1:36772 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:11:27.038924 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:27.037942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.0.98.61:41423)
0927 21:11:27.037985 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:27.037988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:27.038285 (+ 297us) server_negotiation.cc:407] Connection header received
0927 21:11:27.038393 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:27.038394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:27.038406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:27.038436 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:27.038848 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36778: BlockingRecv error: recv got EOF from 127.0.0.1:36778 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:11:27.105011 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:27.104242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.0.98.61:41423)
0927 21:11:27.104310 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:27.104313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:27.104461 (+ 148us) server_negotiation.cc:407] Connection header received
0927 21:11:27.104519 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:27.104533 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:27.104544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:27.104614 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:27.104939 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36790: BlockingRecv error: recv got EOF from 127.0.0.1:36790 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 21:11:27.235000 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:27.234184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.0.98.61:41423)
0927 21:11:27.234271 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:27.234274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:27.234383 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:11:27.234499 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:27.234500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:27.234512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:27.234589 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:27.234936 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36802: BlockingRecv error: recv got EOF from 127.0.0.1:36802 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 21:11:27.492887 22841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:27.492264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.0.98.61:41423)
0927 21:11:27.492339 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:27.492345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:27.492353 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:11:27.492378 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:27.492379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:27.492388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:27.492452 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:27.492797 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36808: BlockingRecv error: recv got EOF from 127.0.0.1:36808 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:11:27.847726 22839 catalog_manager.cc:1259] Loaded cluster ID: 9f612c5057ce45b09d04fad4dba7dee3
I20260927 21:11:27.847793 22839 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: loading cluster ID for follower catalog manager: success
I20260927 21:11:27.848629 22839 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: acquiring CA information for follower catalog manager: success
I20260927 21:11:27.848945 22839 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6118a617dfeb4d21a9377e607b77c1f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:11:27.870328 22854 catalog_manager.cc:1259] Loaded cluster ID: 9f612c5057ce45b09d04fad4dba7dee3
I20260927 21:11:27.870404 22854 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: loading cluster ID for follower catalog manager: success
I20260927 21:11:27.871245 22854 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: acquiring CA information for follower catalog manager: success
I20260927 21:11:27.871632 22854 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b9f310c2967404193ba63a2c9b2eb8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:11:28.008742 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:28.006032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.98.61:41423)
0927 21:11:28.006164 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:28.006166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:28.006202 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:11:28.006307 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:28.006308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:28.006369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:28.006458 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:28.006616 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.007144 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.007925 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.008132 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.008207 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:28.008386 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:28.008493 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:28.008593 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:28.008616 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:11:28.008637 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260927 21:11:28.012897 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:28.010569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.98.60:36733)
0927 21:11:28.010668 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:28.010670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:28.010677 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:11:28.010710 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:28.010711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:28.010744 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:28.010811 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:28.010890 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.011430 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.012100 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.012289 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.012341 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:28.012495 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:28.012597 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:28.012670 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:28.012705 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 21:11:28.012724 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
I20260927 21:11:28.017534 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:28.014782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.0.98.61:41423)
0927 21:11:28.014836 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:28.014838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:28.014928 (+ 90us) server_negotiation.cc:407] Connection header received
0927 21:11:28.014981 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:28.014983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:28.014994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:28.015050 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:28.015213 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.015733 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.016775 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.017024 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.017087 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:28.017230 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:28.017265 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:28.017293 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:28.017438 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 21:11:28.017463 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:11:28.017899 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:28.014695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.98.60:36733)
0927 21:11:28.014734 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:28.014736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:28.014772 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:11:28.015039 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:28.015041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:28.015049 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:28.015141 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:28.015276 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.015977 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.017020 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.017296 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.017363 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:28.017526 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:28.017559 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:28.017581 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:28.017815 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:11:28.017837 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:11:28.017990 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:28.014619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.0.98.62:43469)
0927 21:11:28.014786 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:28.014788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:28.014892 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:11:28.014981 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:28.014988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:28.015043 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:28.015151 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:28.015325 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.016073 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.016811 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:28.017150 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:28.017216 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:28.017443 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:28.017556 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:28.017621 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:28.017854 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:11:28.017874 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":47,"threads_started":1}
I20260927 21:11:28.019755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22603
I20260927 21:11:28.025179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22678
I20260927 21:11:28.031677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22753
2026-09-27T21:11:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2767 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:11:28 dist-test-slave-8bgs krb5kdc[22970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:11:28 dist-test-slave-8bgs krb5kdc[22970](info): set up 2 sockets
Sep 27 21:11:28 dist-test-slave-8bgs krb5kdc[22970](info): commencing operation
krb5kdc: starting...
W20260927 21:11:29.059298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/test-user.keytab.
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 21:11:29.174582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36209,127.0.98.61:44659,127.0.98.60:44153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:11:29.255589 22983 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:11:29.255843 22983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:29.255875 22983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:29.255921 22983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:29.257076 22983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:29.257143 22983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:29.257169 22983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:29.257200 22983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:29.257225 22983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:29.258713 22983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36209,127.0.98.61:44659,127.0.98.60:44153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:29.258994 22983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:29.259249 22983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:29.263909 22989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:29.263904 22991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:29.263913 22988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:29.264140 22983 server_base.cc:1034] running on GCE node
I20260927 21:11:29.264354 22983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:29.264636 22983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:29.265774 22983 hybrid_clock.cc:648] HybridClock initialized: now 1790543489265765 us; error 27 us; skew 500 ppm
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:29.268273 22983 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 21:11:29.268728 22983 webserver.cc:460] Webserver started at http://127.0.98.62:41769/ using document root <none> and password file <none>
I20260927 21:11:29.268955 22983 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:29.269001 22983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:29.269140 22983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:29.270174 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data/instance:
uuid: "f51c3ab3791b4f278606e478983fd672"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "8fbad89d2edad716da2163703475cbd2"
server_key_iv: "55be6d41fcb377bc8460710cb295b829"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.270525 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f51c3ab3791b4f278606e478983fd672"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "8fbad89d2edad716da2163703475cbd2"
server_key_iv: "55be6d41fcb377bc8460710cb295b829"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.271891 22983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:11:29.272562 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:29.272766 22983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:29.272872 22983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal
uuid: "f51c3ab3791b4f278606e478983fd672"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "8fbad89d2edad716da2163703475cbd2"
server_key_iv: "55be6d41fcb377bc8460710cb295b829"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.272944 22983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:29.282831 22983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:29.285028 22983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:29.285187 22983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:29.288944 22983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36209
I20260927 21:11:29.288951 23050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36209 every 8 connection(s)
I20260927 21:11:29.289312 22983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/data/info.pb
I20260927 21:11:29.290295 23051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:29.291894 23051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:29.299240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 22983
I20260927 21:11:29.299343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-0/wal/instance
I20260927 21:11:29.299536 392 external_mini_cluster.cc:1428] Setting key a590f2b704f0fd3cf00b495a1e5fe1f8
I20260927 21:11:29.301090 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.292660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.0.98.62:36209)
0927 21:11:29.292765 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.292769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.292800 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:11:29.293333 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.293335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.293351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.293447 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.293974 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.294781 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.295690 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.295985 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.296110 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.298052 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.298058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.298058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.298065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.299460 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.299855 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.299858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.299858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.299942 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.300166 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.300167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.300167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.300383 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.300479 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.300698 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.300905 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":79,"thread_start_us":38,"threads_started":1}
I20260927 21:11:29.301101 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.292496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36209 (local address 127.0.0.1:47508)
0927 21:11:29.292660 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.292692 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.293164 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.293452 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.293454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.293719 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.293867 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.293871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.294883 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.294884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.295595 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.295597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.296094 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.296095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.296157 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.296607 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.296615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.297952 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.299549 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.299554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.299557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.299774 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.300009 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.300010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.300011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.300101 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.300455 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.300461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.300522 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.300622 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.300822 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260927 21:11:29.301919 23051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:29.302260 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.302141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35902)
0927 21:11:29.302189 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.302231 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:11:29.302389 23002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
I20260927 21:11:29.302783 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.302691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35914)
0927 21:11:29.302738 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.302761 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:11:29.302891 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 21:11:29.321792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44659
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36209,127.0.98.61:44659,127.0.98.60:44153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:11:29.352129 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } attempt: 1
I20260927 21:11:29.352612 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.352478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35928)
0927 21:11:29.352539 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.352581 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:11:29.353036 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.352943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35938)
0927 21:11:29.352989 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.353016 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:11:29.353165 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
W20260927 21:11:29.401402 23058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:11:29.401650 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:29.401681 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:29.401723 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:29.402820 23058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:29.402884 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:29.402908 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:29.402936 23058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:29.402966 23058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:29.404439 23058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36209,127.0.98.61:44659,127.0.98.60:44153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:29.404721 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:29.404939 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:29.409611 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:29.409619 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:29.409632 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:29.409844 23058 server_base.cc:1034] running on GCE node
I20260927 21:11:29.410084 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:29.410393 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:29.411553 23058 hybrid_clock.cc:648] HybridClock initialized: now 1790543489411545 us; error 35 us; skew 500 ppm
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:29.414009 23058 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 21:11:29.414470 23058 webserver.cc:460] Webserver started at http://127.0.98.61:44279/ using document root <none> and password file <none>
I20260927 21:11:29.414705 23058 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:29.414774 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:29.414924 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:29.416086 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data/instance:
uuid: "155513dee3f8490fb7d045418add50b0"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "925ec3a2e31219a6019c06880d8fc99d"
server_key_iv: "786b53cf2991749a320d09acf60f27a7"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.416445 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal/instance:
uuid: "155513dee3f8490fb7d045418add50b0"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "925ec3a2e31219a6019c06880d8fc99d"
server_key_iv: "786b53cf2991749a320d09acf60f27a7"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.417742 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:11:29.418515 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:29.418699 23058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:11:29.418795 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal
uuid: "155513dee3f8490fb7d045418add50b0"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "925ec3a2e31219a6019c06880d8fc99d"
server_key_iv: "786b53cf2991749a320d09acf60f27a7"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.418877 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:29.421315 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } attempt: 2
I20260927 21:11:29.421698 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.421579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35946)
0927 21:11:29.421637 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.421672 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:11:29.422014 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.421923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35958)
0927 21:11:29.421965 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.421989 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:11:29.422106 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44659: connect: Connection refused (error 111)
I20260927 21:11:29.433640 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:29.436163 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:29.436335 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:29.440573 23058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44659
I20260927 21:11:29.440609 23125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44659 every 8 connection(s)
I20260927 21:11:29.440963 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/data/info.pb
I20260927 21:11:29.441913 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:29.443531 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:29.446394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 23058
I20260927 21:11:29.446532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-1/wal/instance
I20260927 21:11:29.446727 392 external_mini_cluster.cc:1428] Setting key b874e988c938338c2bb62ca227a5e3b7
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:29.452950 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.444206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.0.98.62:36209)
0927 21:11:29.444284 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.444287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.444449 (+ 162us) server_negotiation.cc:407] Connection header received
0927 21:11:29.444907 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.444909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.444918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.444990 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.445632 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.446103 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.447301 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.447639 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.447715 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.450035 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.450036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.450037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.450044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.451551 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.451975 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.451977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.451977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.452042 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.452292 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.452293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.452293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.452416 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.452496 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.452774 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.452861 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:11:29.453140 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.444161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36209 (local address 127.0.0.1:47512)
0927 21:11:29.444330 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.444367 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.444802 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.445084 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.445087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.445330 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.445540 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.445544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.446198 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.446199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.447219 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.447222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.447734 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.447735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.447836 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.448370 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.448381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.449940 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.451653 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.451660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.451662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.451898 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.452119 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.452119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.452120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.452227 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.452486 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.452492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.452636 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.452739 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.452905 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260927 21:11:29.453718 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:11:29.461328 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.454042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.98.61:44659)
0927 21:11:29.454214 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.454223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.454242 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:11:29.454263 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.454265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.454277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.454347 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.454624 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.455427 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.456238 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.456631 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.456725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.458456 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.458463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.458463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.458470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.460111 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.460534 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.460537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.460538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.460601 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.460821 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.460822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.460822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.460952 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.461012 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.461189 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.461256 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
I20260927 21:11:29.461467 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.454006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35970)
0927 21:11:29.454080 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.454093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.454147 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.454373 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.454374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.454495 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.454550 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.454551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.455497 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.455498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.456106 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.456108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.456724 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.456725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.456788 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.457203 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.457208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.458378 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.460196 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.460197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.460198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.460442 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.460679 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.460680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.460680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.460753 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.461020 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.461021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.461081 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.461169 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.461342 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":66}
I20260927 21:11:29.462114 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:29.462466 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.462332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38134)
0927 21:11:29.462382 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.462436 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:11:29.462620 23076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
I20260927 21:11:29.463116 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.462968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38138)
0927 21:11:29.463025 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.463067 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:11:29.463231 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44153: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 21:11:29.471196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6L0Aex/build/release/bin/kudu
/tmp/dist-test-task6L0Aex/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44153
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36209,127.0.98.61:44659,127.0.98.60:44153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:11:29.512465 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } attempt: 1
I20260927 21:11:29.512981 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.512832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38146)
0927 21:11:29.512902 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.512947 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 21:11:29.513226 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } attempt: 3
I20260927 21:11:29.513386 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.513269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38150)
0927 21:11:29.513316 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.513345 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:11:29.513484 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44153: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:11:29.520332 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.513487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35976)
0927 21:11:29.513532 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.513542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.513582 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.513745 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.513746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.513921 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.513995 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.513997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.514636 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.514637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.515488 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.515491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.515839 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.515840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.515914 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.516314 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.516319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.517647 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.519296 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.519297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.519298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.519476 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.519688 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.519689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.519689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.519754 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.519997 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.519999 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.520046 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.520133 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.520233 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:11:29.520327 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.513520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.0.98.61:44659)
0927 21:11:29.513556 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.513558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.513572 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:11:29.513649 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.513650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.513661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.513723 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.514069 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.514562 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.515564 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.515767 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.515822 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.517734 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.517735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.517736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.517743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.519203 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.519562 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.519563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.519563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.519622 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.519817 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.519818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.519819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.519931 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.520006 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.520160 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.520235 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:11:29.520745 23051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:29.521126 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.520968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38162)
0927 21:11:29.521063 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.521099 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:11:29.521569 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.521442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38174)
0927 21:11:29.521507 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.521549 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 21:11:29.521667 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44153: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
I20260927 21:11:29.552827 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } attempt: 1
I20260927 21:11:29.553280 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.553144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38178)
0927 21:11:29.553212 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.553252 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:11:29.553653 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.553558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38188)
0927 21:11:29.553602 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.553627 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:11:29.553738 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44153: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
W20260927 21:11:29.557297 23133 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:11:29.557497 23133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:11:29.557514 23133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:11:29.557576 23133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:11:29.558706 23133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:11:29.558743 23133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:11:29.558756 23133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:11:29.558770 23133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:11:29.558806 23133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:11:29.560274 23133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36209,127.0.98.61:44659,127.0.98.60:44153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:11:29.560535 23133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:11:29.560765 23133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:11:29.565904 23139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:29.565912 23138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:11:29.565963 23141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:11:29.566078 23133 server_base.cc:1034] running on GCE node
I20260927 21:11:29.566326 23133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:11:29.566613 23133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:11:29.567777 23133 hybrid_clock.cc:648] HybridClock initialized: now 1790543489567756 us; error 51 us; skew 500 ppm
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:11:29.574831 23133 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 21:11:29.575330 23133 webserver.cc:460] Webserver started at http://127.0.98.60:42479/ using document root <none> and password file <none>
I20260927 21:11:29.575577 23133 fs_manager.cc:359] Metadata directory not provided
I20260927 21:11:29.575644 23133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:11:29.575802 23133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:11:29.576895 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data/instance:
uuid: "faefe9a58896488599ce88d8cd66a599"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "5f348bb4421c09b03cf2f06e619bddd3"
server_key_iv: "b30b98d8d3ab52288fca99b13c34df52"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.577252 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal/instance:
uuid: "faefe9a58896488599ce88d8cd66a599"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "5f348bb4421c09b03cf2f06e619bddd3"
server_key_iv: "b30b98d8d3ab52288fca99b13c34df52"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.578528 23133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:11:29.579268 23148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:11:29.579437 23133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:11:29.579533 23133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data,/tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal
uuid: "faefe9a58896488599ce88d8cd66a599"
format_stamp: "Formatted at 2026-09-27 21:11:29 on dist-test-slave-8bgs"
server_key: "5f348bb4421c09b03cf2f06e619bddd3"
server_key_iv: "b30b98d8d3ab52288fca99b13c34df52"
server_key_version: "encryptionkey@0"
I20260927 21:11:29.579624 23133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:11:29.581630 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } attempt: 2
I20260927 21:11:29.581996 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.581875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38200)
0927 21:11:29.581935 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.581969 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:11:29.582329 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.582252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38212)
0927 21:11:29.582291 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.582314 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 21:11:29.582432 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44153: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44153: connect: Connection refused (error 111)
I20260927 21:11:29.599079 23133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:11:29.601531 23133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:11:29.601697 23133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:11:29.605563 23133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44153
I20260927 21:11:29.605643 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44153 every 8 connection(s)
I20260927 21:11:29.605988 23133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/data/info.pb
I20260927 21:11:29.606992 23201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:11:29.608534 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:29.616039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6L0Aex/build/release/bin/kudu as pid 23133
I20260927 21:11:29.616161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6L0Aex/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790543220945705-392-0/minicluster-data/master-2/wal/instance
I20260927 21:11:29.616369 392 external_mini_cluster.cc:1428] Setting key 751ea19e6836239a16d8da444bb1f7f9
I20260927 21:11:29.617185 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.609192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.0.98.62:36209)
0927 21:11:29.609273 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.609276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.609441 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:11:29.609904 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.609906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.609915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.609968 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.610552 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.611011 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.611946 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.612137 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.612195 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.614159 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.614161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.614161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.614167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.615574 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.616010 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.616011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.616012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.616092 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.616417 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.616419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.616419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.616565 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.616632 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.616987 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.617082 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:11:29.617376 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.609136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36209 (local address 127.0.0.1:47520)
0927 21:11:29.609318 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.609354 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.609802 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.610043 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.610046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.610283 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.610468 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.610472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.611103 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.611104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.611866 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.611867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.612225 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.612225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.612334 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.612758 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.612766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.614077 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.615659 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.615665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.615666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.615919 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.616185 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.616185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.616186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.616344 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.616645 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.616652 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.616832 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.616951 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.617129 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260927 21:11:29.618008 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:11:29.624835 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.618280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44659 (local address 127.0.0.1:35992)
0927 21:11:29.618346 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.618355 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.618390 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.618505 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.618505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.618609 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.618652 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.618653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.619336 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.619337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.619955 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.619957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.620368 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.620368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.620437 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.620749 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.620753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.621968 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.623780 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.623780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.623781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.624039 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.624231 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.624232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.624232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.624292 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.624517 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.624518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.624564 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.624646 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.624732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:11:29.624850 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.618338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.0.98.61:44659)
0927 21:11:29.618384 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.618386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.618395 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:11:29.618439 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.618440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.618449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.618503 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.618712 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.619270 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.620029 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.620280 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.620366 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.622072 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.622074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.622074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.622082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.623692 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.624117 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.624118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.624118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.624168 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.624361 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.624361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.624361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.624455 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.624502 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.624663 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.624737 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:11:29.625196 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:11:29.626196 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.616766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.0.98.60:44153)
0927 21:11:29.616961 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.616965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.616984 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:11:29.617038 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.617040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.617053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.617127 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.617849 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.618779 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.619700 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.620029 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.620098 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.622665 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.622672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.622673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.622679 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.624684 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.625079 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.625082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.625082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.625145 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.625592 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.625593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.625593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.625730 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.625887 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.626005 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.626084 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260927 21:11:29.628892 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } attempt: 2
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 21:11:29.633699 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.625431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38220)
0927 21:11:29.625480 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.625490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.625532 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.625848 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.625848 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.625990 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.626025 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.626027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.626936 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.626937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.627540 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.627543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.627896 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.627897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.627950 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.628270 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.628275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.629822 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.632276 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.632277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.632277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.632484 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.633067 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.633068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.633068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.633137 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.633376 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.633377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.633413 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.633480 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.633571 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:11:29.633795 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.625490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.60:44153)
0927 21:11:29.625690 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.625692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.625699 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:11:29.625716 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.625716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.625724 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.625834 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.626118 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.626713 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.627627 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.627823 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.627879 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.629900 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.629901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.629901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.629907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.632199 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.632577 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.632579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.632579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.632638 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.633204 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.633205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.633205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.633306 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.633384 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.633511 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.633620 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:11:29.635361 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: Bootstrap starting.
I20260927 21:11:29.635556 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.626933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.98.62:36209)
0927 21:11:29.626974 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.626975 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.627003 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:11:29.627024 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.627024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.627033 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.627122 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.627555 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.628155 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.629069 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.629258 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.629306 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.631905 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.631907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.631907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.631914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.633676 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.634435 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.634436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.634436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.634494 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.634700 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.634701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.634701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.634832 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.634997 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.635402 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.635472 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:11:29.636179 23201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:29.636544 23201 log.cc:824] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:29.637400 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.629646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.98.60:44153)
0927 21:11:29.629694 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.629696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.629702 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:11:29.629719 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.629719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.629727 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.629796 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.630103 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.630844 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.632157 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.632520 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.632785 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.634545 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.634546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.634547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.634553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.636157 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.636550 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.636551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.636551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.636648 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.636879 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.636880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.636881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.637034 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.637099 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.637252 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.637319 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:11:29.637416 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.629489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38232)
0927 21:11:29.629541 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.629551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.629593 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.629800 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.629801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.629977 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.630016 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.630018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.630964 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.630965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.631803 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.631806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.632613 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.632614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.632676 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.633026 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.633030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.634462 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.636249 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.636251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.636251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.636466 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.636728 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.636729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.636729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.636791 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.637093 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.637094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.637137 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.637228 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.637318 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:11:29.638114 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: No bootstrap required, opened a new log
I20260927 21:11:29.638882 23051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672: Bootstrap starting.
I20260927 21:11:29.639950 23051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:29.640034 23201 raft_consensus.cc:348] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.640206 23201 raft_consensus.cc:729] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faefe9a58896488599ce88d8cd66a599, State: Initialized, Role: FOLLOWER
I20260927 21:11:29.640275 23051 log.cc:824] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:29.640295 23201 consensus_queue.cc:260] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.640548 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.640607 23201 sys_catalog.cc:564] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:29.640697 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 21:11:29.641010 23051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672: No bootstrap required, opened a new log
W20260927 21:11:29.642701 23220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:29.642763 23220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:29.643187 23051 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.643328 23051 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51c3ab3791b4f278606e478983fd672, State: Initialized, Role: FOLLOWER
I20260927 21:11:29.643416 23051 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.643668 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.643719 23051 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:29.643815 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:29.642887 23133 master_runner.cc:186] Error getting master registration for 127.0.98.62:36209: OK, instance_id { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" instance_seqno: 1790543489287882 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:11:29.645051 23133 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:11:29.647586 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.636298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.98.61:44659)
0927 21:11:29.636348 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.636350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.636397 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:11:29.636455 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.636456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.636466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.636524 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.637067 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.637615 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.638428 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.638707 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.638780 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.641516 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.641517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.641517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.641524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.645853 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.646266 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.646268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.646268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.646330 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.646547 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.646547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.646548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.646679 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.646821 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.647408 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.647488 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:11:29.647616 22983 master_runner.cc:186] Error getting master registration for 127.0.98.61:44659: OK, instance_id { permanent_uuid: "155513dee3f8490fb7d045418add50b0" instance_seqno: 1790543489439412 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:11:29.650429 22983 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:11:29.645536 23233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:29.650648 23233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:29.651885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:11:29.652822 23025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f51c3ab3791b4f278606e478983fd672"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47524
I20260927 21:11:29.652959 23025 raft_consensus.cc:482] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:11:29.653026 23025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:29.653945 23025 raft_consensus.cc:504] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.654212 23025 leader_election.cc:288] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [CANDIDATE]: Term 1 election: Requested vote from peers 155513dee3f8490fb7d045418add50b0 (127.0.98.61:44659), faefe9a58896488599ce88d8cd66a599 (127.0.98.60:44153)
I20260927 21:11:29.654630 23175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f51c3ab3791b4f278606e478983fd672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "faefe9a58896488599ce88d8cd66a599"
I20260927 21:11:29.654757 23175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:29.655845 23175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f51c3ab3791b4f278606e478983fd672 in term 1.
I20260927 21:11:29.656386 23001 leader_election.cc:302] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f51c3ab3791b4f278606e478983fd672, faefe9a58896488599ce88d8cd66a599; no voters:
I20260927 21:11:29.656504 23222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:11:29.656672 23222 raft_consensus.cc:686] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [term 1 LEADER]: Becoming Leader. State: Replica: f51c3ab3791b4f278606e478983fd672, State: Running, Role: LEADER
I20260927 21:11:29.656777 23222 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.657397 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51c3ab3791b4f278606e478983fd672. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.657606 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:29.664971 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.655671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.0.98.62:36209)
0927 21:11:29.655750 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.655752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.655760 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:11:29.655782 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.655783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.655793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.655868 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.657559 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.658301 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.659340 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.659644 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.659714 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.661858 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.661859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.661860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.661868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.663536 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.663961 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.663962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.663963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.664023 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.664354 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.664356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.664356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.664486 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.664569 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.664799 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.664879 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:11:29.667234 23244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:11:29.667469 23244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:11:29.670477 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.655921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.98.60:44153)
0927 21:11:29.655979 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.655981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.659443 (+ 3462us) server_negotiation.cc:407] Connection header received
0927 21:11:29.659537 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.659538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.659548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.659578 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.660311 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.661030 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.663278 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.663577 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.663615 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.666906 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.666907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.666908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.666916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.669060 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.669643 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.669645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.669645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.669700 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.669909 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.669910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.669910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.670034 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.670105 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.670303 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.670383 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:11:29.672300 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.656725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.0.98.61:44659)
0927 21:11:29.656770 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.656772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.665047 (+ 8275us) server_negotiation.cc:407] Connection header received
0927 21:11:29.665140 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.665141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.665151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.665181 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.666009 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.666736 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.667738 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.668026 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.668071 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.669374 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.669375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.669375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.669382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.671121 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.671512 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.671513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.671514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.671586 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.671791 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.671792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.671792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.671918 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.672019 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.672130 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.672210 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:11:29.673583 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } attempt: 3
I20260927 21:11:29.676215 23175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [term 1 FOLLOWER]: Refusing update from remote peer f51c3ab3791b4f278606e478983fd672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:29.676576 23222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:11:29.677985 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51c3ab3791b4f278606e478983fd672. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.678097 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:29.680018 23222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.680126 23222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: This master's current role is: LEADER
Sep 27 21:11:29 dist-test-slave-8bgs krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 21:11:29.680220 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.680251 23245 mvcc.cc:205] Tried to move back new op lower bound from 7334066133711298560 to 7334066133634953216. Current Snapshot: MvccSnapshot[applied={T|T < 7334066133711298560}]
I20260927 21:11:29.680289 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:29.680469 23246 mvcc.cc:205] Tried to move back new op lower bound from 7334066133711298560 to 7334066133634953216. Current Snapshot: MvccSnapshot[applied={T|T < 7334066133711298560}]
I20260927 21:11:29.683475 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.673864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.0.98.60:44153)
0927 21:11:29.673989 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.673991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.673997 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:11:29.674015 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.674016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.674024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.674073 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.674359 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.675130 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.678169 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.678470 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.678541 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.680661 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:11:29.680662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:11:29.680662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:11:29.680669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:11:29.682246 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.682660 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.682661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.682662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.682720 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:11:29.682922 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:11:29.682923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:11:29.682923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:11:29.683050 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:11:29.683143 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:29.683313 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 21:11:29.683387 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 21:11:29.683492 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:29.673822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44153 (local address 127.0.0.1:38250)
0927 21:11:29.673875 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:11:29.673885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:11:29.673917 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:11:29.674092 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:11:29.674093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:11:29.674242 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:11:29.674280 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.674281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.675193 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.675194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.676012 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:11:29.676014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:11:29.678543 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:11:29.678545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:11:29.678607 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:29.679111 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:11:29.679117 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:11:29.680574 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:11:29.682353 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.682354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.682355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.682586 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.682799 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:11:29.682800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:11:29.682800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:11:29.682857 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:11:29.683147 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:11:29.683148 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:11:29.683206 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:11:29.683274 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:11:29.683398 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:11:29.685050 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: Bootstrap starting.
I20260927 21:11:29.685729 23126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: Neither blocks nor log segments found. Creating new log.
I20260927 21:11:29.686066 23126 log.cc:824] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:11:29.687119 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: No bootstrap required, opened a new log
I20260927 21:11:29.687618 23244 catalog_manager.cc:1347] Generated new cluster ID: b84a2ed46b64448e8b0795488b448829
I20260927 21:11:29.687661 23244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:11:29.689060 23126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.689183 23126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155513dee3f8490fb7d045418add50b0, State: Initialized, Role: FOLLOWER
I20260927 21:11:29.689255 23126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } }
I20260927 21:11:29.689569 23126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:11:29.691126 23099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:11:29.691994 23099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [term 1 FOLLOWER]: Refusing update from remote peer f51c3ab3791b4f278606e478983fd672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:11:29.692266 23241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:11:29.692826 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:11:29.692921 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:11:29.694348 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51c3ab3791b4f278606e478983fd672. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.694567 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:29.694660 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.694763 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:29.694505 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.694890 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:11:29.690937 23100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f51c3ab3791b4f278606e478983fd672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "155513dee3f8490fb7d045418add50b0"
I20260927 21:11:29.705590 23265 mvcc.cc:205] Tried to move back new op lower bound from 7334066133711298560 to 7334066133634953216. Current Snapshot: MvccSnapshot[applied={T|T < 7334066133711298560}]
I20260927 21:11:29.705752 23100 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f51c3ab3791b4f278606e478983fd672 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260927 21:11:29.719585 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f51c3ab3791b4f278606e478983fd672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f51c3ab3791b4f278606e478983fd672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } peers { permanent_uuid: "155513dee3f8490fb7d045418add50b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44659 } } peers { permanent_uuid: "faefe9a58896488599ce88d8cd66a599" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44153 } } }
I20260927 21:11:29.719699 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672 [sys.catalog]: This master's current role is: LEADER
I20260927 21:11:29.722878 23244 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:11:29.723629 23244 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:11:29.733500 23244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f51c3ab3791b4f278606e478983fd672: Generated new TSK 0
I20260927 21:11:29.733693 23244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:11:29.794795 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.787694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.0.98.62:36209)
0927 21:11:29.787734 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.787736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.794031 (+ 6295us) server_negotiation.cc:407] Connection header received
0927 21:11:29.794149 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.794150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.794163 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.794202 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.794720 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47542: BlockingRecv error: recv got EOF from 127.0.0.1:47542 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:11:29.796267 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.787453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.0.98.61:44659)
0927 21:11:29.787501 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.787504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.795058 (+ 7554us) server_negotiation.cc:407] Connection header received
0927 21:11:29.795618 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.795619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.795631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.795683 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.796193 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36010: BlockingRecv error: recv got EOF from 127.0.0.1:36010 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 21:11:29.796733 23205 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.788173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.0.98.60:44153)
0927 21:11:29.788215 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.788217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.795719 (+ 7502us) server_negotiation.cc:407] Connection header received
0927 21:11:29.795745 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.795747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.795758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.795803 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.796670 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38260: BlockingRecv error: recv got EOF from 127.0.0.1:38260 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:11:29.799482 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.798602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.0.98.61:44659)
0927 21:11:29.798645 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.798647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.798880 (+ 233us) server_negotiation.cc:407] Connection header received
0927 21:11:29.798997 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.798998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.799017 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.799047 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.799431 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36018: BlockingRecv error: recv got EOF from 127.0.0.1:36018 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 21:11:29.802665 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.801874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.0.98.61:44659)
0927 21:11:29.801921 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.801922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.802165 (+ 243us) server_negotiation.cc:407] Connection header received
0927 21:11:29.802259 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.802260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.802271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.802300 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.802614 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36030: BlockingRecv error: recv got EOF from 127.0.0.1:36030 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:11:29.806653 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.805899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.0.98.61:44659)
0927 21:11:29.805935 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.805937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.806165 (+ 228us) server_negotiation.cc:407] Connection header received
0927 21:11:29.806261 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.806262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.806272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.806301 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.806607 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36040: BlockingRecv error: recv got EOF from 127.0.0.1:36040 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 21:11:29.812898 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.812078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.0.98.61:44659)
0927 21:11:29.812114 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.812116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.812351 (+ 235us) server_negotiation.cc:407] Connection header received
0927 21:11:29.812446 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.812447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.812458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.812487 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.812848 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36046: BlockingRecv error: recv got EOF from 127.0.0.1:36046 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:11:29.823511 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.822408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.0.98.61:44659)
0927 21:11:29.822454 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.822457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.822788 (+ 331us) server_negotiation.cc:407] Connection header received
0927 21:11:29.822907 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.822909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.822922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.822956 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.823431 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36052: BlockingRecv error: recv got EOF from 127.0.0.1:36052 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:11:29.842053 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.841020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.98.61:44659)
0927 21:11:29.841066 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.841069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.841372 (+ 303us) server_negotiation.cc:407] Connection header received
0927 21:11:29.841496 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.841497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.841510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.841548 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.841990 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36068: BlockingRecv error: recv got EOF from 127.0.0.1:36068 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:11:29.876261 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.875528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.0.98.61:44659)
0927 21:11:29.875607 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.875609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.875700 (+ 91us) server_negotiation.cc:407] Connection header received
0927 21:11:29.875739 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.875740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.875753 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.875847 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.876203 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36082: BlockingRecv error: recv got EOF from 127.0.0.1:36082 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 21:11:29.942236 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:29.941432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.98.61:44659)
0927 21:11:29.941506 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:29.941509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:29.941624 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:11:29.941710 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:29.941728 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:29.941739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:29.941805 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:29.942156 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36088: BlockingRecv error: recv got EOF from 127.0.0.1:36088 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 21:11:30.072203 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:30.071424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.61:44659)
0927 21:11:30.071506 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.071508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.071621 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:11:30.071691 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.071692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.071704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.071782 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:30.072126 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36092: BlockingRecv error: recv got EOF from 127.0.0.1:36092 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 21:11:30.330116 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:11:30.329443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.0.98.61:44659)
0927 21:11:30.329514 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.329517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.329525 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:11:30.329554 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.329555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.329564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.329625 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:11:30.330026 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36104: BlockingRecv error: recv got EOF from 127.0.0.1:36104 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:11:30.643220 23220 catalog_manager.cc:1259] Loaded cluster ID: b84a2ed46b64448e8b0795488b448829
I20260927 21:11:30.643328 23220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: loading cluster ID for follower catalog manager: success
I20260927 21:11:30.644279 23220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: acquiring CA information for follower catalog manager: success
I20260927 21:11:30.644660 23220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P faefe9a58896488599ce88d8cd66a599: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:11:30.693341 23262 catalog_manager.cc:1259] Loaded cluster ID: b84a2ed46b64448e8b0795488b448829
I20260927 21:11:30.693409 23262 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: loading cluster ID for follower catalog manager: success
I20260927 21:11:30.694229 23262 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: acquiring CA information for follower catalog manager: success
I20260927 21:11:30.694548 23262 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 155513dee3f8490fb7d045418add50b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:11:30.846215 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:30.843351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.98.61:44659)
0927 21:11:30.843474 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.843476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.843568 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:11:30.843650 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.843651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.843709 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.843799 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:30.844024 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.844529 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.845417 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.845610 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.845682 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:30.845865 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:30.845976 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:30.846068 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:30.846091 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:11:30.846112 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260927 21:11:30.850630 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:30.848374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.0.98.60:44153)
0927 21:11:30.848487 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.848489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.848498 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:11:30.848552 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.848552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.848587 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.848668 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:30.848814 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.849278 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.849939 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.850123 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.850171 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:30.850291 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:30.850382 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:30.850438 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:30.850492 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 21:11:30.850507 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260927 21:11:30.854806 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:30.852367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.0.98.60:44153)
0927 21:11:30.852410 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.852412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.852460 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:11:30.852560 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.852561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.852572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.852601 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:30.852803 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.853495 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.854231 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.854468 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.854526 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:30.854605 (+ 79us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:30.854636 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:30.854683 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:30.854718 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 21:11:30.854735 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:11:30.855497 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:30.852144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.98.62:36209)
0927 21:11:30.852253 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.852255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.852433 (+ 178us) server_negotiation.cc:407] Connection header received
0927 21:11:30.852514 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.852515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.852561 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.852660 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:30.852818 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.853402 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.854419 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.854728 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.854787 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:30.855058 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:30.855194 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:30.855237 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:30.855367 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:11:30.855405 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
I20260927 21:11:30.855829 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:11:30.852171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.0.98.61:44659)
0927 21:11:30.852288 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 21:11:30.852291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:11:30.852541 (+ 250us) server_negotiation.cc:407] Connection header received
0927 21:11:30.852564 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:11:30.852565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:11:30.852575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:11:30.852636 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:11:30.852759 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.853485 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.854832 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:11:30.855121 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:11:30.855196 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:11:30.855403 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:11:30.855438 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:11:30.855513 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:11:30.855734 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 21:11:30.855758 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 21:11:30.857155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 22983
I20260927 21:11:30.862421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 23058
I20260927 21:11:30.867022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6L0Aex/build/release/bin/kudu with pid 23133
2026-09-27T21:11:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2835 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5603 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269928 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST